I have a Japanese products UTF-8 data-positioned flat file. In my process I have a Data Process shape that endodes the file using a Windows-1252 codepage. I was told that this shape would replace Japanese characters to question marks.
However, this does not appear to work. The result being that hidden whitespace characters are embedded in the file. Then when a Business Rules shape attempts to map profile elements to rule functions, the entire file fails because some rows, containing whitespace characters, have elements that are no longer located at their expected starting position, causing a MANDATORY_EMPTY_ELEMENT error.
How may I strip out these offending characters so that my file matches its profile?