Publishing Huge data to scene7

[asn11]

Hi Team,

We are using CQ5.6.1

Our prod instance has got close to 30 GB of assets, and we are looking for ways to publish this huge data to scene7.

For starters we tried with close to 2 GB of assets in one of the lower environments.

 -It took 5 hours for data to get published to scene7.

- Though the data was around 2 GB, the CQ server went on occupying space exponentially.

- For publishing 2 GB of data the CQ instance space utilization grew by alarmingly atleast 18-20 GBs.

Why is it so?

We do not have importer (scene7 to DAM) enabled.

All we did was selecting a high level folder and click on publish to scene7.

It would be very helpful if I am guided how to deal with such huge data.

[Daniel_Hollerun]

I think this is a great question; one that I'd like to hear a response. However, I don't think it is a question of Target but rather Experience Manager.

Would it make sense to ask it here instead?

http://help-forums.adobe.com/content/adobeforums/en/experience-manager-forum/adobe-experience-manage...

[smacdonald2008]

I passed this question to the S7 team. We should hear back soon. 

[Jitendra_S_Toma]

@asn_177,

This is the one reason, we should look for data storage/Cloud for large amount of data. Keeping everything in our repository is always problematic in such cases.

Jitendra

[asn11]

Thank You Scott and Daniel for passing to right direction.

@Jitendra,

We do not have any cloud storage available in current setup.

Would want to know if its possible at all, with just CQ 5.6.1 and Scene7 account for company.

If yes,could you please point us to right direction.

Regards,

[Jitendra_S_Toma]

Sorry man. If cloud storage isn't there, I don't think it would be right to proceed with this at this point of time. It was just a thought that in these situations, cloud storage could have a better option.

Here is one doc which talks about configuring Amazon S3 storage with CQ5.6.1

http://cq-ops.tumblr.com/post/69692603146/how-to-host-cq-data-store-on-amazon-s3

Jitendra

[Kunal_Gaba_]

My hunch is that when the scene7 publish job runs on DAM assets it writes some status metadata property on each asset to track the status of the publish operation. And if there are huge number of assets then the number of writes to the repository increases proportionately which leads to creation of new tarpm tar files on the file system. (As tar file is never overwritten, new data is always appended)

I guess if you run the tarpm optimization job after the scene7 publish is complete then you should be able to reclaim the extra redundant disk space which was consumed during the publishing. This is my guess but there could be other reasons for the growth during the publishing. 

https://helpx.adobe.com/experience-manager/kb/analyze-unusual-repository-growth.html

https://helpx.adobe.com/experience-manager/kb/TarPMOptimization.html

[mvance · Accepted solution]

The Scene7 uploader in 5.6.1 sends files one a time, so this explains that it takes a long time to go through the content to the Scene7 servers.

You may check the job logs in your SPS instance to see if there are any errors or warnings logged.

I don't have an answer about the space utilization topic