Discussion:
Error With HTTPClient and JsonParser escaped chars
(too old to reply)
Gabriel Labarrera
2020-12-03 23:58:02 UTC
Permalink
hi,

I'm having an error with a rest service return when the response have escaped chars the connector fails to parse the result.

Response:

{
"dc_grupo": "Mapa de Procesos \"Lector\"",
"id_grupo": "5250"
}

Error:

{
"status": "fail",
"connectorname": "ObtenerDatosUsuario",
"operation": "lookup",
"exception": "java.lang.Exception: com.ibm.commons.util.io.json.parser.TokenMgrError: Lexical error at line 1, column 1216. Encountered: "\n" (10), after : "\""",
"message": "com.ibm.commons.util.io.json.parser.TokenMgrError: Lexical error at line 1, column 1216. Encountered: "\n" (10), after : "\""",
"class": "java.lang.Exception"
}

Is this a know issue on TDI?
Jens Thomassen
2020-12-08 01:46:32 UTC
Permalink
Hi Gabriel,
I am not able to reproduce this.
Could you please give some more details about how you have configured your AssemblyLine?
The title seems to indicate that you used the HTTP Client Connector with an attached JSON Parser, but when I try using
SDI 7.2 with the latest fixpack, making sure I receive the response you specified, I get no error.
If you are indeed using this setup, could you for test purposes remove the JSON Parser, and log the conn Entry e.g.
with code like
task.dumpEntry(conn)
in the "GetNext Successful" hook of the "ObtenerDatosUsuario" connector.
That should allow us to see exactly what is received by the connector.
Jens Thomassen
2020-12-08 04:03:49 UTC
Permalink
Hi Gabriel,
looking more closely at your error message, I believe that the problem is not due to escaped characters,
it happens because the HTTP server that you get the response from has added random newlines in the JSON
string constants, maybe because it felt that a line of more than 1216 characters was too long.
I guess this because the error happened on line 1, column 1216.
Or maybe some string constant actually started with a newline, but instead of encoding it using "\n",
the server just output an actual newline.

Here is an example of what I think happens, but with much shorter lines (max 12 characters),
to make it easier to see the issue.
Instead of
{
"dc_grupo": "Mapa de Procesos \"Lector\"",
"id_grupo": "5250"
}

You get something like
{ "dc_grupo": "
Mapa de Proc
esos \"Lector
\"", "id_grupo"
: "5250"}

Obviously this is incorrect JSON, and cannot easily be parsed.
But with SDI you can do something like this:
Remove the JSON Parser from the "ObtenerDatosUsuario" connector.
In the "After GetNext" hook, add these two lines:
body = conn.getString("http.bodyAsString").replace("\n", "")
conn.merge(conn.fromJSON(body))
The first line gets the entire HTTP body and removes all newlines.
The second line parses the body as JSON and merges the result into the conn Entry.
Feel free to improve the script as needed.
Gabriel Labarrera
2020-12-15 15:39:27 UTC
Permalink
Hi Jens,

Indeed this was the problem and replace the "\n" was the solution.
Post by Jens Thomassen
Hi Gabriel,
looking more closely at your error message, I believe that the problem is not due to escaped characters,
it happens because the HTTP server that you get the response from has added random newlines in the JSON
string constants, maybe because it felt that a line of more than 1216 characters was too long.
I guess this because the error happened on line 1, column 1216.
Or maybe some string constant actually started with a newline, but instead of encoding it using "\n",
the server just output an actual newline.
Here is an example of what I think happens, but with much shorter lines (max 12 characters),
to make it easier to see the issue.
Instead of
{
"dc_grupo": "Mapa de Procesos \"Lector\"",
"id_grupo": "5250"
}
You get something like
{ "dc_grupo": "
Mapa de Proc
esos \"Lector
\"", "id_grupo"
: "5250"}
Obviously this is incorrect JSON, and cannot easily be parsed.
Remove the JSON Parser from the "ObtenerDatosUsuario" connector.
body = conn.getString("http.bodyAsString").replace("\n", "")
conn.merge(conn.fromJSON(body))
The first line gets the entire HTTP body and removes all newlines.
The second line parses the body as JSON and merges the result into the conn Entry.
Feel free to improve the script as needed.
Continue reading on narkive:
Loading...