CVP 8.5(1) data is crossing between calls

Issue with Custom element 7008 node: We are trying to figure out why CVP appears to be skipping over parts of a custom coded element on about 1% (or less) of the calls.  The element takes an account number (element data) and queries a web service to get details on that account.  The element then takes values from that web service and attempts to load that information into element data.  Although it fails to load the data into element data, it returns as successful/found in the log.  We can see that the data is coming from the web service successfully based on the logging.  See below.

------------------------------------------------------

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,enter,

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,doAction

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,Call Id: 083D21EA9AC511E68C920022915819D8

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,serviceEndpoint: services01.us.xxxxxxx.net:443

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,In call service method...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,Account Number: 9817

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,Setting Service Endpoint: https://services01.us.xxxxxxx.net:443/WS/Deposits/DepositAccount/1.0

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,Setting HTTP Headers...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,Setting 2way authentication...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,GetAccountRequestType...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,GetAccountRequestDocument...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,setGetAccountRequest...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.714,7008,custom,Info,GetAccountResponseDocument...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,getGetAccountResponseType...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,getXMLText from respType...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,New Util...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,--------------------------------------------------

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,<xml-fragment xsi:type="dep:GetAccountSuccessResponseType" xsi:schemaLocation="http://xxxxxxx.com/xsd/msg/ent/deposits/depositaccount DepositAccount_Msg_v1_0.xsd" xmlns:err="http://xxxxxxx.com/xsd/lib/common/error" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ent="http://xxxxxxx.com/xsd/lib/ent" xmlns:dep="http://xxxxxxx.com/xsd/msg/ent/deposits/depositaccount" xmlns:acc="http://xxxxxxx.com/xsd/lib/ent/deposits/depositaccount/account" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:NS1="http://schemas.xmlsoap.org/soap/envelope/"><dep:isSuccess>true</dep:isSuccess><acc:depositAccount><ent:corporationNumber>10002</ent:corporationNumber><acc:accountNumber>*******************9817</acc:accountNumber><acc:productCode>DDA</acc:productCode><acc:subProductCode>80</acc:subProductCode><acc:depositAccountStatusCode>99</acc:depositAccountStatusCode></acc:depositAccount></xml-fragment>

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,--------------------------------------------------

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,Check for Success...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,Get account record lookup success

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,Parsing response data...

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,corporateNumber: 10002

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,productCode: DDA

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,subProductCode: 80

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.776,7008,custom,Info,accountStatus: 99

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.823,7008,element,warning,A substitution representing Element data from element "7008" and variable named "PRODUCT_CODE" referred to non-existent information. An empty string was substituted instead.

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.823,7008,element,warning,A substitution representing Element data from element "7008" and variable named "CORPORATION_NUMBER" referred to non-existent information. An empty string was substituted instead.

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.823,7008,element,warning,A substitution representing Element data from element "7008" and variable named "LAST_FOUR_ACCOUNT_NUMBER" referred to non-existent information. An empty string was substituted instead.

32.21.166.116.1477494583477.1626.IVR_Ret_Main,10/26/2016 11:10:57.823,7008,exit,RecordFound

Looks like spelling mismatch. productCode vs. PRODUCT_CODE

11:10:57.776,7008,custom,Info,productCode: DDA

element,warning,A substitution representing Element data from element

"7008" and variable named "PRODUCT_CODE"

No, there is no misspelling , productCode is just what I typed in the code to describe the variable data in the log.

PRODUCT_CODE is the element name.  This code works 99% of the time. We don't see the problem in our QA environment, just in production where the load is much higher.

The "DDA" string that was returned from the web service is not being loaded into element data.  This element is also setting some of the element data to session data in the data tab of the element.  That is when the below error shows up in the log:

'element,warning,A substitution representing Element data from element

"7008" and variable named "PRODUCT_CODE"'

I would recommend to move to a supported version and check, if the problem persists and is performance related and you feel its not custom code issue then open a TAC case.