Stuck In Solution Document for Website Slowness and Call 10000 commerce products in AEM

[sunitaBorn]

Dear All,

I have one use case where I need to display more than 10000 products from my commercetools in the AEM Product listing page. So how can I achieve this ? I have below couple of questions in my mind. Can anyone please help me here.

1) Where these images of 10000 products will be stored ? Is It inside AEM Asset "or" Inside dynamic media ? 

2) We will configure these commercetools in AEM and get the details of the product and so every time, do we need to hit the commercetools API to get the product details ?

I have another issue where customer is hitting the website www.abc.com and this website is very slow. What all the thing I need to debug to see why the AEM website is very slow ?

[EmmaWalsh2]

To display 10,000+ products from commercetools in AEM, you need efficient image storage and data retrieval.

Where to store images?

AEM Assets: Full control but not ideal for high volume.
Dynamic Media: Best for performance, uses Adobe's CDN.
External CDN: commercetools, AWS S3, or Cloudinary for scalability.

Do we need to call commercetools API every time?

Direct API calls: Slow, not ideal.
Cache in AEM: Store product data in AEM for better speed.
Hybrid approach: Cache frequently accessed data, update via cron jobs or webhooks.

[kapil_rajoria · Accepted solution]

Hi @sunitaBorn ,
1) Where these images of 10000 products will be stored ? Is It inside AEM Asset "or" Inside dynamic media ? 

AEM Assets

Store the product images in AEM Assets if:

• You want to manage the images directly in AEM.

• You need to apply DAM (Digital Asset Management) features like metadata, versioning, and workflows.

• The number of images is manageable within AEM's storage limits.

• Pros:

  • Tight integration with AEM.

  • Easy to manage and organize assets.

• Cons:

  • AEM Assets may not scale well for very large datasets (e.g., 10,000+ images).

  • Performance may degrade if too many assets are stored in AEM.

Dynamic Media

Use Dynamic Media if:

• You need to optimize images for different devices and screen sizes (e.g., responsive images).

• You want to offload image storage and delivery to a dedicated service.

• You have a large number of images and need scalable delivery.

• Pros:

  • Scalable and optimized for large datasets.

  • Supports on-the-fly image transformations (e.g., resizing, cropping).

  • Reduces the load on AEM.

• Cons:

  • Requires additional configuration and integration with AEM.

  • May involve additional costs for Dynamic Media usage.

Recommendation:

• Use Dynamic Media for storing and delivering product images, especially for a large number of products. It is designed for scalability and performance.

2) We will configure these commercetools in AEM and get the details of the product and so every time, do we need to hit the commercetools API to get the product details ?

Hitting the commercetools API for every request can lead to severe performance issues which will slow the website and for a large number of products, if API calls handled improperly, also presents the risk of server downtown and crash.

Solution:

1. use of GraphQL:

It allows you to request only the data you need, reducing the payload size and improving performance. commercetools provides native support for GraphQL, making it an ideal choice for fetching product data.

Benefits of GraphQL:

• Efficient Data Fetching: Request only the required fields (e.g., product name, price, image URL) instead of fetching the entire product object.

• Reduced Payload: Smaller response sizes compared to REST APIs.

• Single Endpoint: Simplifies API integration by using a single endpoint for all queries.

2. use of Adobe I/O with a cache layer:

Adobe I/O provides a robust platform for integrating AEM with external systems like commercetools. You can use Adobe I/O Runtime (serverless functions) to create a middleware layer that handles API calls, caching, and data transformation.

Benefits of Adobe I/O:

• Serverless Architecture: Scalable and cost-effective.

• Event-Driven: Trigger actions (e.g., data sync) based on events.

• Integration with AEM: Seamlessly connect AEM with external APIs.

Must thing for any implementation - Adding a Cache Layer

A cache layer is critical for improving performance and reducing the load on commercetools. You can implement caching at multiple levels.

Cache Layer Options:

1. AEM Dispatcher Cache:

   • Cache the product listing page at the dispatcher level.

   • Invalidate the cache when product data is updated.

2. In-Memory Cache:

   • Use an in-memory cache (e.g., Redis or AEM's built-in cache) to store product data.

   • Example: Cache product data for 1 hour.

3. CDN Cache:

   • Use a Content Delivery Network (CDN) to cache product images and API responses.

3) I have another issue where customer is hitting the website www.abc.com and this website is very slow. What all the thing I need to debug to see why the AEM website is very slow ?

You can follow the following debugging steps:
1. Open the website in chrome and right click and click on inspect. Go to the Lighthouse tab and click on Analyze Page Load. You will see the Performance, Accessibility, Best Practices and SEO scores along with all the details why they are less. Try to solve all of them and make them rise above 80.
2. Resolve all console errors.

[AmitVishwakarma]

1. Where to Store 10,000+ Product Images:

       - Use Dynamic Media for large product image datasets (10,000+ images). It offers scalability, optimized delivery, and on-the-fly image transformations.
       - AEM Assets is suitable for smaller datasets but may not scale well with large numbers of images.

2. Fetching Product Details:

        - Avoid API calls on every page load.
         - Use GraphQL to fetch only required product data (reducing payload size).
Implement a cache layer:
         - AEM Dispatcher Cache for product listing pages.
         - In-memory cache (e.g., Redis) for faster access to product data.
         - CDN Cache for images and API responses.