Discussion:
TDI 7.2 : How to get decompressed data from HTTP response.
(too old to reply)
DIVYA RAIBAGKAR
2020-10-12 13:08:52 UTC
Permalink
Hi ,
I am currently working on a custom REST API connector. I am trying to get authentication token from a POST request.
It uses following headers while requesting
Content-Length: 140
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: /
Accept-Language: en;q=1
Cache-Control: no-cache, no-store, max-age=0, must-revalidate

I do get a 200 OK response with "http.Content-Encoding": "gzip" and the http.body/http.bodyAsString is in the compressed format.
Do I have to decompress the content explicitly ?
If yes how can I do that in TDI AL?
Tobias Ernstberger
2020-10-12 19:48:06 UTC
Permalink
Hi,

typically you would use TDI HTTP Client without compression - so no Accept-Encoding Heading in the request.
Then everything should work fine directly.

If you really need compression (e.g. for limited bandwidth or very large result objects), you have to decompress with custom code in a TDI hook.
If you offer gzip and deflate in the request, your result from the server might be plain, gzip or deflate. I recommend using a hook before input mapping like "After CallReply" (in CallReply Mode) to do the decompression.
You have so evaluate the content of conn["http.Content-Encoding"] to choose the right decompression. For GZIP, here an example:

var byteArrayInputStream = new java.io.ByteArrayInputStream(conn["http.bodyAsBytes"].getValue(0));
var gzipInputStream = new java.util.zip.GZIPInputStream(byteArrayInputStream);
var inputStreamReader = new java.io.InputStreamReader(gzipInputStream, java.nio.charset.StandardCharsets.UTF_8)
var bufferedReader = new java.io.BufferedReader(inputStreamReader)
var output = new java.lang.StringBuilder();

var line="";
while((line = bufferedReader.readLine()) != null){
output.append(line);
}
task.logmsg(output.toString());

Be aware that there are some pitfalls like the encoding (UTF-8 in my example) and maybe more. I didn't test this extensively, just made a quick and dirty test.
In summary: If you can avoid the compression, I would recommend that. Otherwise, use my template to build a robust solution.
For further help, you can contact IBM Security Expert Labs.

Kind regards,
Tobias Ernstberger
DIVYA RAIBAGKAR
2020-10-13 08:24:12 UTC
Permalink
Post by Tobias Ernstberger
Hi,
typically you would use TDI HTTP Client without compression - so no Accept-Encoding Heading in the request.
Then everything should work fine directly.
If you really need compression (e.g. for limited bandwidth or very large result objects), you have to decompress with custom code in a TDI hook.
If you offer gzip and deflate in the request, your result from the server might be plain, gzip or deflate. I recommend using a hook before input mapping like "After CallReply" (in CallReply Mode) to do the decompression.
var byteArrayInputStream = new java.io.ByteArrayInputStream(conn["http.bodyAsBytes"].getValue(0));
var gzipInputStream = new java.util.zip.GZIPInputStream(byteArrayInputStream);
var inputStreamReader = new java.io.InputStreamReader(gzipInputStream, java.nio.charset.StandardCharsets.UTF_8)
var bufferedReader = new java.io.BufferedReader(inputStreamReader)
var output = new java.lang.StringBuilder();
var line="";
while((line = bufferedReader.readLine()) != null){
output.append(line);
}
task.logmsg(output.toString());
Be aware that there are some pitfalls like the encoding (UTF-8 in my example) and maybe more. I didn't test this extensively, just made a quick and dirty test.
In summary: If you can avoid the compression, I would recommend that. Otherwise, use my template to build a robust solution.
For further help, you can contact IBM Security Expert Labs.
Kind regards,
Tobias Ernstberger
Thank you Tobias for your response.
Since I have to use third party APIs I cannot modify the headers to be sent in request.
Though with your input I am able to decompress the response data.

For additional information,
When we use Postman client or browser to send the request with Accept-Encoding : gzip,deflate.
The received response is not compressed and shows plain text data in body with response header Content-Encoding:gzip.
But its not the case in TDI.

Do you know the reason behind this?
Tobias Ernstberger
2020-10-13 11:46:49 UTC
Permalink
Post by DIVYA RAIBAGKAR
Post by Tobias Ernstberger
Hi,
typically you would use TDI HTTP Client without compression - so no Accept-Encoding Heading in the request.
Then everything should work fine directly.
If you really need compression (e.g. for limited bandwidth or very large result objects), you have to decompress with custom code in a TDI hook.
If you offer gzip and deflate in the request, your result from the server might be plain, gzip or deflate. I recommend using a hook before input mapping like "After CallReply" (in CallReply Mode) to do the decompression.
var byteArrayInputStream = new java.io.ByteArrayInputStream(conn["http.bodyAsBytes"].getValue(0));
var gzipInputStream = new java.util.zip.GZIPInputStream(byteArrayInputStream);
var inputStreamReader = new java.io.InputStreamReader(gzipInputStream, java.nio.charset.StandardCharsets.UTF_8)
var bufferedReader = new java.io.BufferedReader(inputStreamReader)
var output = new java.lang.StringBuilder();
var line="";
while((line = bufferedReader.readLine()) != null){
output.append(line);
}
task.logmsg(output.toString());
Be aware that there are some pitfalls like the encoding (UTF-8 in my example) and maybe more. I didn't test this extensively, just made a quick and dirty test.
In summary: If you can avoid the compression, I would recommend that. Otherwise, use my template to build a robust solution.
For further help, you can contact IBM Security Expert Labs.
Kind regards,
Tobias Ernstberger
Thank you Tobias for your response.
Since I have to use third party APIs I cannot modify the headers to be sent in request.
Though with your input I am able to decompress the response data.
For additional information,
When we use Postman client or browser to send the request with Accept-Encoding : gzip,deflate.
The received response is not compressed and shows plain text data in body with response header Content-Encoding:gzip.
But its not the case in TDI.
Do you know the reason behind this?
Glad to hear that you made progress. I assume, that Postman does the decompression before displaying the result. The response header clearly shows that the result was compressed. This is typical for all http clients supporting compression natively - which seems to be not the case for TDI. Luckily, TDI is flexible enough to add it :)


Regards, Tobias

Continue reading on narkive:
Loading...