Is your form an XDP or PDF? If so, did you create it in Designer from within Workbench (i.e. is the Form Bridge or the Process Fields on the form)?

I suspect the process type is set to "short-lived", instead of "long-lived". In Workbench, right-click on the process and select "Configure...". Navigate to the Advanced tab. Under Run Time Settings, ensure the Type is set to "long-lived". Save and re-deploy the process.

You are correct. Forms is needed to render an xfa-based document. Note that you can create a custom Render process that will merge the xml data with a PDF form using the importData operation of the FormDataIntegration service.

The expressions that work in your test environment suggest that the "Store form data as xdp" setting has been removed in the XML variable. When this setting is ON the data within the XML variable will always execute XPath expressions as namespace unaware (as in your development environment). This setting was done so that valid XPath expressions for the data will always be generated regardless if there are namespace aliases specified in the process properties.Verify that setting is on in the XML ...

Are you using IE 8? If so, do you have this patch installed http://www.microsoft.com/technet/security/bulletin/MS09-054.mspx?/ck