Upload a file to xtk:fileRes via SOAP or other HTTP request?

[yetanother_andrey]

Here is the deal: we don't have anything in ACC for any kind of DevOps or CI/CD stuff. But we have a possibility to do SOAP calls and several other JSP endpoints are accessible also. Frankly better than nothing.

But! There is one task which seem to be impossible to have automated without injecting custom schema and JS library: upload a file to the instance and link it to the xtk:fileRes resource.

What I've discovered so far:

1. What Client console does?

One can use traceviewer to see which calls Client Console performs. Unfortunately, this is what it shows right after Console checks whether the file already exists and before calling PostUpload:

In other words: nothing helpful. Using traffic sniffer it became clear that Client Console actually does the same that all WebApps do to upload a file, see (2) below.

2. WebApp could also upload a file to the instance disk. How it works?

It's pretty simple and straightforward. In fact, when uploading using WebApp, the browser sends POST request to uploadFile.jsp, which, in return, responds with a bit of HTML and JS code, where all the details of the uploaded file are included. But to use this for automation tools, one will have to a) perform the request with required fields inside, b) parse the output. This second part is not so easy because you really have to parse HTML, get the JS from it and then parse JS. (Or you can use regexes to grab the strings, but it's a duct tape, really)

3. Inject custom stuff into ACC to have your automation working

For sure, custom schema can be built exposing custom SOAP method, and custom JS can be written to handle this method and use ACC JS API (like File()) to store uploaded file on disk. But, no. Good automation is the automation which might be thrown away at any time with no need to change anything in the target system. In ideal world, target system should know nothing about automation system. (unless target system doesn't have built-in APIs for external management)

Short conclusion: there is no easy way to pick a file from your local drive and put in into ACC without Client Console.

Adobe, would you please be so kind and share your thoughts on this? Maybe there is a solution hidden somewhere without proper documentation?

Or maybe anyone has a solution? Please?

[Jonathon_wodnicki]

Hi,

Follow these steps:

1. POST the file to /nl/jsp/uploadFile.jsp, using content type multipart/form-data. Ignore the response, there's nothing useful in it.
2. Call SOAPAction: xtk:fileRes#PostUpload with this entity content: <fileRes md5="<file's md5 hash>" originalName="<original filename since it becomes md5 on disk>" useMd5AsFilename="true" publish="true" storageType="5" xtkschema="xtk:fileRes" xmlns="urn:xtk:fileRes" />

Might need to create the xtk:fileRes ahead of the PostUpload call using xtk:persist#Write, give it a shot.

Thanks,

-Jon

[yetanother_andrey]

This does not work, or I'm missing something. Below is my code in python.

import hashlib
import requests

# session = requests session, with applied authentication (X-Security-Token header and cookie)

ac_url = ''  # ac base url https://hostname.domain

# prepare file
file_name = 'some_ext_logo.png'
file_contents = None
file_md5 = None
with open(file_name, 'rb') as fp:
    file_contents = fp.read()
file_md5 = hashlib.md5(file_contents).hexdigest()

# create xtk:fileRes
create_fileRes_payload = f"""
<SOAP-ENV:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ns='urn:xtk:persist' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>
    <SOAP-ENV:Body>
        <Write xmlns='urn:xtk:persist' SOAP-ENV:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/'>
            <__sessiontoken xsi:type='xsd:string'></__sessiontoken>
            <domDoc xsi:type='ns:Element' SOAP-ENV:encodingStyle='http://xml.apache.org/xml-soap/literalxml'>
                <fileRes _operation="insertOrUpdate" md5="{file_md5}" codepage="1252" folder-id="964523908"  label="{file_name}" storageType="5" useMd5AsFilename="true" xtkschema="xtk:fileRes" internalName="testFileMd5" _key="@internalName"/>
            </domDoc>
        </Write>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>
"""
create_fileRes_headers = {
    'Content-Type': 'application/soap+xml',
    'action': 'xtk:persist#Write',
    'SOAPAction': 'xtk:persist#Write'
}
create_fileRes_response = session.post(f"{ac_url}/nl/jsp/soaprouter.jsp", data=create_fileRes_payload, headers=create_fileRes_headers)
create_fileRes_response.raise_for_status()
print(create_fileRes_response.text)

# push file to ac
push_file_response = session.post(f"{ac_url}/nl/jsp/uploadFile.jsp", files={'file': (f"{file_md5}.png", file_contents, 'image/png')})
push_file_response.raise_for_status()
print(push_file_response.text)

# post upload
post_upload_payload = f"""
<SOAP-ENV:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ns='urn:xtk:fileRes' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>
    <SOAP-ENV:Body>
        <PostUpload xmlns='urn:xtk:fileRes' SOAP-ENV:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/'>
            <__sessiontoken xsi:type='xsd:string'></__sessiontoken>
            <entity xsi:type='ns:Element' SOAP-ENV:encodingStyle='http://xml.apache.org/xml-soap/literalxml'>
                <fileRes md5="{file_md5}" originalName="D:\Documents\Seafile\B2CMA\Dev\per task\librarian upload file\{file_name}" publish="true" storageType="5" useMd5AsFilename="true" xtkschema="xtk:fileRes" />
            </entity>
            <domParams xsi:type='ns:Element' SOAP-ENV:encodingStyle='http://xml.apache.org/xml-soap/literalxml'>
                <params />
            </domParams>
        </PostUpload>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>
"""
post_upload_headers = {
    'Content-Type': 'application/soap+xml',
    'action': 'xtk:fileRes#PostUpload',
    'SOAPAction': 'xtk:fileRes#PostUpload'
}
post_upload_response = session.post(f"{ac_url}/nl/jsp/soaprouter.jsp", data=post_upload_payload, headers=post_upload_headers)
post_upload_response.raise_for_status()
print(post_upload_response.text)

xtk:fileRes is created without any issues with usual response from Write method:

<?xml version='1.0'?>
<SOAP-ENV:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ns='urn:wpp:default' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>
    <SOAP-ENV:Body>
        <WriteResponse xmlns='urn:wpp:default' SOAP-ENV:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/'></WriteResponse>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

push file also works (you have to use md5 as a file name if useMd5AsFilename=true):

Ok
<html xmlns="http://www.w3.org/1999/xhtml" >
  <head>
    <script type="text/javascript">
      if( window.parent && window.parent.document.controller &&
          typeof(window.parent.document.controller.uploadFileCallBack) == "function" )
      {
        var aFilesInfo = new Array();

            aFilesInfo.push({paramName : "file",
                             fileName : "65e5f1e8970e5de6446011d3e1228fd6.png",
                             newFileName : "null",
                             md5 : "null"});
        window.parent.document.controller.uploadFileCallBack(aFilesInfo)
      }
    </script>
  </head>
  <body>
  </body>
</html>

Even post upload seems to work:

<?xml version='1.0'?>
<SOAP-ENV:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ns='urn:xtk:fileRes' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>
    <SOAP-ENV:Body>
        <PostUploadResponse xmlns='urn:xtk:fileRes' SOAP-ENV:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/'>
            <entity xsi:type='ns:Element' SOAP-ENV:encodingStyle='http://xml.apache.org/xml-soap/literalxml'>
                <fileRes fileName="/usr/local/neolane/nl6/var/res/customer_mkt_dev1/65e5f1e8970e5de6446011d3e1228fd6.png" md5="65e5f1e8970e5de6446011d3e1228fd6" originalName="D:\Documents\Seafile\B2CMA\Dev\per task\librarian upload file\some_ext_logo.png" publish="false" storageType="5" useMd5AsFilename="true" xtkschema="xtk:fileRes" />
            </entity>
        </PostUploadResponse>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

But atthe end console shows this:

So the file was not published correctly and is not available.

[Jonathon_wodnicki]

All that looks reasonable.

Ok here's where this gets weird:

The console makes a *third* call to re-re-write the fileRes metadata, after the PostUpload call.

In that third call it sets loaded="1":

<fileRes _operation="update" codepage="0" contentType="image/png" fileName="/usr/local/neolane/nl6/var/res/local/9f09fadd381938586f4872435d26631a.png" id="375379" loaded="1" md5="9f09fadd381938586f4872435d26631a" name="kittens.png" userContentType="0" xtkschema="xtk:fileRes"/>

Which is a calculated expression in schema so shouldn't be set manually:

    <!-- calculated fields -->
    <attribute desc="The file is loaded on the server" expr="iif(@storageType=0 or @storageType=1, iif(@md5='' or @MD5='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',0,1), iif(@fileName='',0,1))"
               label="File loaded" name="loaded" type="boolean"/>

Since @storageType is 5 here, check in the Campaign console that fileName is set for the row?

Thanks,

-Jon

[yetanother_andrey]

Yes, console does this. But this does not work as expected outside the console.

Console does this after PostUpload:

<SOAP-ENV:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ns='urn:xtk:persist' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>
    <SOAP-ENV:Body>
        <Write xmlns='urn:xtk:persist' SOAP-ENV:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/'>
            <__sessiontoken xsi:type='xsd:string'></__sessiontoken>
            <domDoc xsi:type='ns:Element' SOAP-ENV:encodingStyle='http://xml.apache.org/xml-soap/literalxml'>
                <fileRes _operation="update" codepage="0" contentType="application/x-javascript" fileName="/usr/local/neolane/nl6/var/res/customer_mkt_dev1/sec.js" id="964763428" loaded="1" md5="cc3d7f57d9f47e3c9a28f27385ea030b" name="sec.js" userContentType="20" xtkschema="xtk:fileRes">
                    <history _operation="insert" activityType="2" description="Updating file 'D:\Documents\Seafile\B2CMA\Dev\blackhole\sec.js' on the server" id="964763488" lastModified="2020-09-30 12:38:12.590Z" md5="cc3d7f57d9f47e3c9a28f27385ea030b" modifiedBy-id="361091" xtkschema="xtk:fileResHistory">
                        <modifiedBy _cs="" />
                    </history>
                </fileRes>
            </domDoc>
        </Write>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

I've added similar call to my python script:

last_fileRes_update_payload = f"""
<SOAP-ENV:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ns='urn:xtk:persist' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>
    <SOAP-ENV:Body>
        <Write xmlns='urn:xtk:persist' SOAP-ENV:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/'>
            <__sessiontoken xsi:type='xsd:string'></__sessiontoken>
            <domDoc xsi:type='ns:Element' SOAP-ENV:encodingStyle='http://xml.apache.org/xml-soap/literalxml'>
                <fileRes _operation="update" codepage="0" contentType="image/png" fileName="/usr/local/neolane/nl6/var/res/customer_mkt_dev1/{file_md5}.png" loaded="1" md5="{file_md5}" name="{file_name}" userContentType="20" internalName="testFileMd5" _key="@internalName" xtkschema="xtk:fileRes">
                    <history _operation="insert" activityType="2" description="Updating file 'D:\Documents\Seafile\B2CMA\Dev\per task\librarian upload file\{file_name}' on the server" md5="{file_md5}" xtkschema="xtk:fileResHistory"/>
                </fileRes>
            </domDoc>
        </Write>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>
"""
last_fileRes_update_headers = {
    'Content-Type': 'application/soap+xml',
    'action': 'xtk:persist#Write',
    'SOAPAction': 'xtk:persist#Write'
}
last_fileRes_update_response = session.post(f"{ac_url}/nl/jsp/soaprouter.jsp", data=last_fileRes_update_payload, headers=last_fileRes_update_headers)
print(last_fileRes_update_response.text)

And got response:

<?xml version='1.0'?>
<SOAP-ENV:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ns='urn:wpp:default' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>
    <SOAP-ENV:Body>
        <WriteResponse xmlns='urn:wpp:default' SOAP-ENV:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/'></WriteResponse>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Furthermore, I see that file _was_ uploaded and stored on disk:

-rw-r--r-- 1 neolane neolane 950 Oct 22 08:59 /usr/local/neolane/nl6/var/res/customer_mkt_dev1/65e5f1e8970e5de6446011d3e1228fd6.png

And also, @fileName is populated with correct full file path.

But in the console:

If "Publish now" is clicked, console sends this request:

<?xml version='1.0'?>
<SOAP-ENV:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:ns='urn:xtk:fileRes' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>
    <SOAP-ENV:Body>
        <PublishIfNeeded xmlns='urn:xtk:fileRes' SOAP-ENV:encodingStyle='http://schemas.xmlsoap.org/soap/encoding/'>
            <__sessiontoken xsi:type='xsd:string'></__sessiontoken>
            <param xsi:type='xsd:element'>
                <fileRes _cs="some_ext_logo.png" alt="" codepage="0" contentType="image/png" fileName="/usr/local/neolane/nl6/var/res/customer_mkt_dev1/65e5f1e8970e5de6446011d3e1228fd6.png" height="0" id="982178670" internalName="testFileMd5" label="some_ext_logo.png" loaded="1" md5="65e5f1e8970e5de6446011d3e1228fd6" name="" nature="" originalName="" publish="0" storageType="5" useMd5AsFilename="1" userContentType="0" version="" width="0" />
            </param>
        </PublishIfNeeded>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

And it fails!

<?xml version='1.0'?>
<SOAP-ENV:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'>
    <SOAP-ENV:Body>
        <SOAP-ENV:Fault>
            <faultcode>SOAP-ENV:Server</faultcode>
            <faultstring xsi:type='xsd:string'>SOP-330011 Error while executing the method 'PublishIfNeeded' of service 'xtk:fileRes'.</faultstring>
            <detail xsi:type='xsd:string'>XSV-350059 Unable to upload the images to the tracking servers.</detail>
        </SOAP-ENV:Fault>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

[Sukrity_Wadhwa]

Hi @yetanother_andrey,

Were you able to resolve this query with the given solution? Do let us know.

Thanks!

[yetanother_andrey]

Nope