Discussion:
Bad performance in the HTTP server connector
(too old to reply)
p***@itadvisor.se
2017-08-11 11:39:41 UTC
Permalink
Raw Message
Hello,

I am having problems with poor performance when using the (SDI 7.2) HTTP Server Connector.

I am using ISAM to access the service and it always takes between 4-6 seconds when a request is made to it.

Here's an example from the ibmdi.log where SDI is just getting a simple HEAD request from ISAM (to check that SDI is still alive) and all the AL does is to return a 200 as http.status:

2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS310I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is started.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS967I AssemblyLine started by oidc-userinfo.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS258I AssemblyLine AssemblyLines/OIDC-Userinfo is started in manual mode.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS311I -- add runtime Connector: HTTPServer.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDIS498I Using provided parameter.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDJP314I Client connection closed.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS088I Finished iterating.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS315I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is stopped.

I have turned off all logging, increased memory using the JVM parameters -Xms and -Xmm, but still the same.

Any ideas what I can do to improve the response times of the connector?

//Pär
Franzw
2017-08-14 09:07:14 UTC
Permalink
Raw Message
Post by p***@itadvisor.se
Hello,
I am having problems with poor performance when using the (SDI 7.2) HTTP Server Connector.
I am using ISAM to access the service and it always takes between 4-6 seconds when a request is made to it.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS310I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is started.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS967I AssemblyLine started by oidc-userinfo.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS258I AssemblyLine AssemblyLines/OIDC-Userinfo is started in manual mode.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS311I -- add runtime Connector: HTTPServer.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDIS498I Using provided parameter.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDJP314I Client connection closed.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS088I Finished iterating.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS315I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is stopped.
I have turned off all logging, increased memory using the JVM parameters -Xms and -Xmm, but still the same.
Any ideas what I can do to improve the response times of the connector?
//Pär
Hi Pär,

This very much sounds to me like a DNS timeout.

Is this by chance a machine with mulitple network interfaces - if so you should be able to solve it by ensuring TDI only binds to the relevant interface. I am not remembering exactly how to do this if all network interfaces are to be available - but I seem to remember that it has been discussed here earlier.
Anyhow - if this is a DNS issue there will be a lot of solutions anyhow.

Check this first - if that is not the problem come back...

Regards
Franz Wolfhagen
Jason Williams
2017-08-14 17:19:41 UTC
Permalink
Raw Message
Post by Franzw
Post by p***@itadvisor.se
Hello,
I am having problems with poor performance when using the (SDI 7.2) HTTP Server Connector.
I am using ISAM to access the service and it always takes between 4-6 seconds when a request is made to it.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS310I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is started.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS967I AssemblyLine started by oidc-userinfo.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS258I AssemblyLine AssemblyLines/OIDC-Userinfo is started in manual mode.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS311I -- add runtime Connector: HTTPServer.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDIS498I Using provided parameter.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDJP314I Client connection closed.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS088I Finished iterating.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS315I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is stopped.
I have turned off all logging, increased memory using the JVM parameters -Xms and -Xmm, but still the same.
Any ideas what I can do to improve the response times of the connector?
//Pär
Hi Pär,
This very much sounds to me like a DNS timeout.
Is this by chance a machine with mulitple network interfaces - if so you should be able to solve it by ensuring TDI only binds to the relevant interface. I am not remembering exactly how to do this if all network interfaces are to be available - but I seem to remember that it has been discussed here earlier.
Anyhow - if this is a DNS issue there will be a lot of solutions anyhow.
Check this first - if that is not the problem come back...
Regards
Franz Wolfhagen
Found in the solution.properties, the following parameters are the ones used to bind the TDI Server to the IP

## The properties determine the default bind address and the remote bind address for the Server API.
## * means bind to all network interfaces. The Remote Bind Address overrides the Default one.
## Only one IP address should be set. No hostnames are accepted.
## Mind that the java.rmi.server.hostname property is set implicitly to equal the Remote Bind Address property when used.
##This will cause the client stubs to create sockets on the specified Remote Bind Address.
# com.ibm.di.default.bind.address=*
# api.remote.bind.address=*
Pär Kidman
2017-08-17 13:40:52 UTC
Permalink
Raw Message
Post by Jason Williams
Post by Franzw
Post by p***@itadvisor.se
Hello,
I am having problems with poor performance when using the (SDI 7.2) HTTP Server Connector.
I am using ISAM to access the service and it always takes between 4-6 seconds when a request is made to it.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS310I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is started.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS967I AssemblyLine started by oidc-userinfo.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS258I AssemblyLine AssemblyLines/OIDC-Userinfo is started in manual mode.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS311I -- add runtime Connector: HTTPServer.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDIS498I Using provided parameter.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDJP314I Client connection closed.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS088I Finished iterating.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS315I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is stopped.
I have turned off all logging, increased memory using the JVM parameters -Xms and -Xmm, but still the same.
Any ideas what I can do to improve the response times of the connector?
//Pär
Hi Pär,
This very much sounds to me like a DNS timeout.
Is this by chance a machine with mulitple network interfaces - if so you should be able to solve it by ensuring TDI only binds to the relevant interface. I am not remembering exactly how to do this if all network interfaces are to be available - but I seem to remember that it has been discussed here earlier.
Anyhow - if this is a DNS issue there will be a lot of solutions anyhow.
Check this first - if that is not the problem come back...
Regards
Franz Wolfhagen
Found in the solution.properties, the following parameters are the ones used to bind the TDI Server to the IP
## The properties determine the default bind address and the remote bind address for the Server API.
## * means bind to all network interfaces. The Remote Bind Address overrides the Default one.
## Only one IP address should be set. No hostnames are accepted.
## Mind that the java.rmi.server.hostname property is set implicitly to equal the Remote Bind Address property when used.
##This will cause the client stubs to create sockets on the specified Remote Bind Address.
# com.ibm.di.default.bind.address=*
# api.remote.bind.address=*
Hello!

Yes the server has two network interfaces, one for applications and another for backups so I therefore followed your recommendation and updated the solution.properties to ensure that the server is listening on the correct interface (which in this case is 10.122.111.217). Using netstat I can also verify that the server has bound on the correct interface:

TCP 10.122.111.217:1298 0.0.0.0:0 LISTENING
TCP 10.122.111.217:8989 0.0.0.0:0 LISTENING

Interesting is that if I access the HTTP server from a browser on the same network (without going through ISAM) I get response times on approx 10ms...... Neither do I get poor performance when testing it through ISAM on my local development environment. I was thinking that perhaps there was some routing issue but then again why does the HTTP server then respond immediately and then sits still for 4-5 secs before it returns the response? In that case wouldn't the initial request also be slow?

If you have any ideas, please feel free to share!
Franzw
2017-08-19 15:35:49 UTC
Permalink
Raw Message
Post by Pär Kidman
Post by Jason Williams
Post by Franzw
Post by p***@itadvisor.se
Hello,
I am having problems with poor performance when using the (SDI 7.2) HTTP Server Connector.
I am using ISAM to access the service and it always takes between 4-6 seconds when a request is made to it.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS310I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is started.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS967I AssemblyLine started by oidc-userinfo.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS258I AssemblyLine AssemblyLines/OIDC-Userinfo is started in manual mode.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS311I -- add runtime Connector: HTTPServer.
2017-08-11 12:59:41,892 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDIS498I Using provided parameter.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - [HTTPServer] CTGDJP314I Client connection closed.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS088I Finished iterating.
2017-08-11 12:59:46,419 INFO [AssemblyLine.AssemblyLines/OIDC-Userinfo] - CTGDIS315I AssemblyLine worker thread: AssemblyLines/OIDC-Userinfo.HTTPServer.21 is stopped.
I have turned off all logging, increased memory using the JVM parameters -Xms and -Xmm, but still the same.
Any ideas what I can do to improve the response times of the connector?
//Pär
Hi Pär,
This very much sounds to me like a DNS timeout.
Is this by chance a machine with mulitple network interfaces - if so you should be able to solve it by ensuring TDI only binds to the relevant interface. I am not remembering exactly how to do this if all network interfaces are to be available - but I seem to remember that it has been discussed here earlier.
Anyhow - if this is a DNS issue there will be a lot of solutions anyhow.
Check this first - if that is not the problem come back...
Regards
Franz Wolfhagen
Found in the solution.properties, the following parameters are the ones used to bind the TDI Server to the IP
## The properties determine the default bind address and the remote bind address for the Server API.
## * means bind to all network interfaces. The Remote Bind Address overrides the Default one.
## Only one IP address should be set. No hostnames are accepted.
## Mind that the java.rmi.server.hostname property is set implicitly to equal the Remote Bind Address property when used.
##This will cause the client stubs to create sockets on the specified Remote Bind Address.
# com.ibm.di.default.bind.address=*
# api.remote.bind.address=*
Hello!
TCP 10.122.111.217:1298 0.0.0.0:0 LISTENING
TCP 10.122.111.217:8989 0.0.0.0:0 LISTENING
Interesting is that if I access the HTTP server from a browser on the same network (without going through ISAM) I get response times on approx 10ms...... Neither do I get poor performance when testing it through ISAM on my local development environment. I was thinking that perhaps there was some routing issue but then again why does the HTTP server then respond immediately and then sits still for 4-5 secs before it returns the response? In that case wouldn't the initial request also be slow?
If you have any ideas, please feel free to share!
It doesn't sound like a routing problem - it sounds like a host name resolution problem. if routing fails it will not work - DNS on the other side will go through the chain of DNS servers and has a timeout.

The quick fix in many cases is to use the hosts file to avoid DNS - this is not a good long time solution, but may work depending on your setup. TDI has some quirks in DNS name resolution that I never fully understood when there are more than one network interface - so even with the settings in solution.properties there might be a bug lurking...

HTH
Regards
Franz Wolfhagen

Loading...