Errors in delivery logs mid workflow in hybrid model - campaign classic version 7 gs11

[balakrishnavalavala]

Hi Team,

     We are getting below errors in mid delivery logs workflow.

Could some one please help ASAP to help in our Root Cause Analysis. We have search in internet with below error strings especially with "FOH2" protocal but no where we got the information even in single website. It looks like related to pure Adobe/mid issue without any information in internet.

Hence requesting to help ASAP and seems like issues is recurring in our marketing instance causing delays to business.

Please confirm if any other information needed for better analysis from your end on this to get the correct solutions.

Awaiting for the solution..

Line 26412: 2021-11-23 15:34:56.286-05 00013950 000144A4 1 error log FDOH-910001 FOH2: The connection is closed. (iRc=16384)
Line 26413: 2021-11-23 15:34:56.286-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_12_1')' could not be executed. (iRc=16384)
Line 26414: 2021-11-23 15:34:56.286-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_12_1')' could not be executed. (iRc=16384)
Line 26415: 2021-11-23 15:34:56.630-05 00013950 000144A4 1 error log INT-150012 The HTTP query returned a 'Internal Server Error' type error (500) (iRc=-53)
Line 26416: 2021-11-23 15:34:56.630-05 00013950 000144A4 1 error log FDOH-910001 FOH2: The connection is closed. (iRc=16384)
Line 26417: 2021-11-23 15:34:56.630-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_11_1')' could not be executed. (iRc=16384)
Line 26418: 2021-11-23 15:34:56.630-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_11_1')' could not be executed. (iRc=16384)
Line 26419: 2021-11-23 15:34:56.989-05 00013950 000144A4 1 error log INT-150012 The HTTP query returned a 'Internal Server Error' type error (500) (iRc=-53)
Line 26420: 2021-11-23 15:34:56.989-05 00013950 000144A4 1 error log FDOH-910001 FOH2: The connection is closed. (iRc=16384)
Line 26421: 2021-11-23 15:34:56.989-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_10_1')' could not be executed. (iRc=16384)
Line 26422: 2021-11-23 15:34:56.989-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_10_1')' could not be executed. (iRc=16384)

Line 26569: 2021-11-23 15:35:01.111-05 00013950 000144A4 1 error log INT-150012 The HTTP query returned a 'Internal Server Error' type error (500) (iRc=-53)
Line 26570: 2021-11-23 15:35:01.111-05 00013950 000144A4 1 error log FDOH-910001 FOH2: The connection is closed. (iRc=16384)
Line 26571: 2021-11-23 15:35:01.111-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_15_1')' could not be executed. (iRc=16384)
Line 26572: 2021-11-23 15:35:01.111-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_15_1')' could not be executed. (iRc=16384)
Line 26576: 2021-11-23 15:35:01.564-05 00013950 000144A4 1 error log INT-150012 The HTTP query returned a 'Internal Server Error' type error (500) (iRc=-53)
Line 26577: 2021-11-23 15:35:01.564-05 00013950 000144A4 1 error log FDOH-910001 FOH2: The connection is closed. (iRc=16384)
Line 26578: 2021-11-23 15:35:01.564-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_14_1')' could not be executed. (iRc=16384)
Line 26579: 2021-11-23 15:35:01.564-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_14_1')' could not be executed. (iRc=16384)
Line 26581: 2021-11-23 15:35:01.908-05 00013950 000144A4 1 error log INT-150012 The HTTP query returned a 'Internal Server Error' type error (500) (iRc=-53)
Line 26582: 2021-11-23 15:35:01.923-05 00013950 000144A4 1 error log FDOH-910001 FOH2: The connection is closed. (iRc=16384)
Line 26583: 2021-11-23 15:35:01.923-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_9_2')' could not be executed. (iRc=16384)
Line 26584: 2021-11-23 15:35:01.923-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_9_2')' could not be executed. (iRc=16384)
Line 26592: 2021-11-23 15:35:02.314-05 00013950 000144A4 1 error log INT-150012 The HTTP query returned a 'Internal Server Error' type error (500) (iRc=-53)
Line 26593: 2021-11-23 15:35:02.314-05 00013950 000144A4 1 error log FDOH-910001 FOH2: The connection is closed. (iRc=16384)
Line 26594: 2021-11-23 15:35:02.314-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_2_2')' could not be executed. (iRc=16384)
Line 26595: 2021-11-23 15:35:02.314-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SELECT DropTableIfExists('wkf485401256_2_2')' could not be executed. (iRc=16384)
Line 26599: 2021-11-23 15:35:02.658-05 00013950 000144A4 1 error log INT-150012 The HTTP query returned a 'Internal Server Error' type error (500) (iRc=-53)
Line 26600: 2021-11-23 15:35:02.673-05 00013950 000144A4 1 error log FDOH-910001 FOH2: The connection is closed. (iRc=16384)
Line 26601: 2021-11-23 15:35:02.673-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SET ENABLE_SEQSCAN to off' could not be executed. (iRc=16384)
Line 26602: 2021-11-23 15:35:02.673-05 00013950 000144A4 1 error log WDB-200001 SQL statement 'SET ENABLE_SEQSCAN to off' could not be executed. (iRc=16384)
Line 26616: 2021-11-23 15:35:04.335-05 00013950 000144A4 1 verbose log WDBC pool (nms:extAccount:FDA_MID prod): 1 connection(s) released, 0 left

Thanks

[david--garcia]

You have a better chance at contacting Adobe Support and raise it as a P1

It seems you have system A trying to communicate with system B and there is an issue where the request is being dropped.

The HTTP query returned a 'Internal Server Error' type error (500) (iRc=-53)

Seems like an access rights/ authentication/<ip><instance> whitelisting issue between the two systems.

[balakrishnavalavala]

Hi David,

    Thank you so much for the helpful information.

From you response I have below thoughts.

1) This internal server error is not coming regularly.

2) As per our observation, we have noticed this error when ever the Mid sourcing log wf crossed 500k records. But as I said even though it crossed 500k, sometimes(max times) wf is running without error. In the error day, This error continuously came for 3 to 6 hours and then it automatically resolved by itself and ran without error.

3) Even I thought the same before that this is some network issue and based on your response it is clear that issue is with networking between Marketing and mid instances. But could you please help how we can proceed to debug the connection loses.

4) We have raised on this issue with Adobe support many times but didn't get the solution and then decided to post here. At present one ticket is in progress already.

Below are my latest questions/thoughts.

1) I have realized that FOH2 means (FDA OVER HTTP PROTOCOL2). We haven't found any information about FOH2 in internet. Hence could someone provide the documentation/links to learn more about FOH2 related to Adobe campaign.

2) We usually see the FOH2 errors when ever the opened/initiated FOH2 connection crosses 30 mins between marketing and mid through mid wf.

3) Hence question is, will there be any timeout setup available in FOH2 process and if yes shall we increase this timeout so that we will eliminate the slowness in update statements that crosses 30 mins.

Thanks

[david--garcia]

try setting NmsMidSourcing_LogsPeriodHour to 1 hour, restart workflow after that change by doing "Unconditional stop" first and then start it again.

When using FDA over HTTPs connection, the Mid-sourcing (delivery logs) (defaultMidSourcingLog) workflow was stuck in the timeframe set by the NmsMidSourcing_LogsPeriodHour option. This would prevent records from being updated with data that occurred after this set timeframe. (NEO-30833)

NmsMidSourcing_LogsPeriodHour

Allows a maximum period (expressed in hours) to be specified as to limit the number of broadlogs recovered every time the synchronization workflow is executed..

[balakrishnavalavala]

Hi David,

    Unfortunately we set the above option with 1 by default and using for so long (years) with this value as 1(And as we know this is the minimum value that we can set for integer data type).

Is there any possibility to set NmsMidSourcing_LogsPeriodHour to 30 mins, so that we will get less number of records to process?

Hence looks like this 1 hour suggestion is not helpful in our case.

Thanks

Thanks

[balakrishnavalavala]

Hi David, 

     I think this statement(When using FDA over HTTPs connection, the Mid-sourcing (delivery logs) (defaultMidSourcingLog) workflow was stuck in the timeframe set by the NmsMidSourcing_LogsPeriodHour option. This would prevent records from being updated with data that occurred after this set timeframe. (NEO-30833)) is talking about the option "NmsMidSourcing_LastBroadLog_defaultEmailMid" instead of NmsMidSourcing_LogsPeriodHour since option "NmsMidSourcing_LastBroadLog_defaultEmailMid" talks about timeframe.(it contains last processed time to collect data for 1 hour after this time) and time can be stuck here by not getting updated with new time (old time + 1 hour).

Please correct me if my understanding is wrong.

And in which way above patch (NEO-30833) fix the issue I posted and why Adobe provided this patch(just want to know the reason for this patch development).

Thanks

[david--garcia]

perhaps @Vipul_Raghav can provide some details as he works closely with the engineering team.

[balakrishnavalavala]

Hi All,

    Could you please help in understanding the purpose of below highlighted parameters in screenshots.

 

  

Thanks