Discussion:
Passing JAVA HashMap object between two scriptconnectors
(too old to reply)
Sergey P
2019-07-21 07:39:24 UTC
Permalink
Hello guys

Excuse me for question that maybe has asked before, but I'm not found question described exactly like my case now.

I build adapter for IGI.
So, in short, I need to pass some Map object between two connectors that both defined in Feed on Iterator Mode. How can I do this?

I can also insert to one ScriptConnector some java code to writing data in file and then read from a file inside another ScriptConnector, but if I could to use more graceful way to implement this logic I'll glad to hear (read) :)

Thank you in advance!

Sergey P.
Eddie Hartman
2019-07-24 09:26:29 UTC
Permalink
Post by Sergey P
Hello guys
Excuse me for question that maybe has asked before, but I'm not found question described exactly like my case now.
I build adapter for IGI.
So, in short, I need to pass some Map object between two connectors that both defined in Feed on Iterator Mode. How can I do this?
I can also insert to one ScriptConnector some java code to writing data in file and then read from a file inside another ScriptConnector, but if I could to use more graceful way to implement this logic I'll glad to hear (read) :)
Thank you in advance!
Sergey P.
Ok, Sergey, as with most things TDI, there are several ways to flay this feline (aka skin the cat :)

One way is to pass stuff around using Input/Output Maps. However, your Iterators only have Input Maps for returning data. No easy way to pass in anything but string values via the parameters.

So instead, you could use the Op Entry which is owned by the AL. The Op Entry is created when the AL starts and it contains any instructions to the AL itself: settings, Connector parameters, AL Operation, etc. However, it is a valid Entry object and is around as long as the AL is running, so we can use it to pass around data within the AL.

I have created an example AL here: https://github.ibm.com/eddie-hartman/_test

To easily use this TDI Project you should:
1. Create a new Project named '_test'
2. Pull down the _test repo - either with 'git clone' or downloading the zip
Put this folder somewhere outside your TDI workspace.
3. Now copy the contents of this _test folder to the _test Project folder
in your TDI workspace.
4. Right-click on your _test Project in the TDI CE Navigator pane and select Refresh

If you just want the 'scriptedIteratorsPassingJavaObject' AL itself, then grab the _test/AssemblyLines/scriptedIteratorsPassingJavaObject.assemblyline file and drag-drop it into your own Project in the TDI CE. You can drag-drop most of the files found in the TDI Project (*.script, *.parser, *.attributemap, and so forth).

Now you should see the scriptedIteratorsPassingJavaObject AL there. Running it you should have both Iterators returning the same four Entries.

If you look in the Connection > Script for the first one you'll see it creates a HashMap and populates it with JS Objects. It then grabs the Op Entry from the AL (task) and saves the map as its '$map' attribute. This is all done in selectEntries().

In the second Iterator the selectEntries grabs the '$map' attribute from the AL's Op Entry and uses it for its own iteration.

Hope this helps! Please let me know if anything is unclear.

/Eddie
Sergey P
2019-07-24 12:44:37 UTC
Permalink
Post by Eddie Hartman
Post by Sergey P
Hello guys
Excuse me for question that maybe has asked before, but I'm not found question described exactly like my case now.
I build adapter for IGI.
So, in short, I need to pass some Map object between two connectors that both defined in Feed on Iterator Mode. How can I do this?
I can also insert to one ScriptConnector some java code to writing data in file and then read from a file inside another ScriptConnector, but if I could to use more graceful way to implement this logic I'll glad to hear (read) :)
Thank you in advance!
Sergey P.
Ok, Sergey, as with most things TDI, there are several ways to flay this feline (aka skin the cat :)
One way is to pass stuff around using Input/Output Maps. However, your Iterators only have Input Maps for returning data. No easy way to pass in anything but string values via the parameters.
So instead, you could use the Op Entry which is owned by the AL. The Op Entry is created when the AL starts and it contains any instructions to the AL itself: settings, Connector parameters, AL Operation, etc. However, it is a valid Entry object and is around as long as the AL is running, so we can use it to pass around data within the AL.
I have created an example AL here: https://github.ibm.com/eddie-hartman/_test
1. Create a new Project named '_test'
2. Pull down the _test repo - either with 'git clone' or downloading the zip
Put this folder somewhere outside your TDI workspace.
3. Now copy the contents of this _test folder to the _test Project folder
in your TDI workspace.
4. Right-click on your _test Project in the TDI CE Navigator pane and select Refresh
If you just want the 'scriptedIteratorsPassingJavaObject' AL itself, then grab the _test/AssemblyLines/scriptedIteratorsPassingJavaObject.assemblyline file and drag-drop it into your own Project in the TDI CE. You can drag-drop most of the files found in the TDI Project (*.script, *.parser, *.attributemap, and so forth).
Now you should see the scriptedIteratorsPassingJavaObject AL there. Running it you should have both Iterators returning the same four Entries.
If you look in the Connection > Script for the first one you'll see it creates a HashMap and populates it with JS Objects. It then grabs the Op Entry from the AL (task) and saves the map as its '$map' attribute. This is all done in selectEntries().
In the second Iterator the selectEntries grabs the '$map' attribute from the AL's Op Entry and uses it for its own iteration.
Hope this helps! Please let me know if anything is unclear.
/Eddie
Hello Eddie.
Thank you a lot for this example and explanation!!!
It's work fine!
Thank you!
Sergey P.
Franzw
2019-07-31 12:06:45 UTC
Permalink
Post by Sergey P
Post by Eddie Hartman
Post by Sergey P
Hello guys
Excuse me for question that maybe has asked before, but I'm not found question described exactly like my case now.
I build adapter for IGI.
So, in short, I need to pass some Map object between two connectors that both defined in Feed on Iterator Mode. How can I do this?
I can also insert to one ScriptConnector some java code to writing data in file and then read from a file inside another ScriptConnector, but if I could to use more graceful way to implement this logic I'll glad to hear (read) :)
Thank you in advance!
Sergey P.
Ok, Sergey, as with most things TDI, there are several ways to flay this feline (aka skin the cat :)
One way is to pass stuff around using Input/Output Maps. However, your Iterators only have Input Maps for returning data. No easy way to pass in anything but string values via the parameters.
So instead, you could use the Op Entry which is owned by the AL. The Op Entry is created when the AL starts and it contains any instructions to the AL itself: settings, Connector parameters, AL Operation, etc. However, it is a valid Entry object and is around as long as the AL is running, so we can use it to pass around data within the AL.
I have created an example AL here: https://github.ibm.com/eddie-hartman/_test
1. Create a new Project named '_test'
2. Pull down the _test repo - either with 'git clone' or downloading the zip
Put this folder somewhere outside your TDI workspace.
3. Now copy the contents of this _test folder to the _test Project folder
in your TDI workspace.
4. Right-click on your _test Project in the TDI CE Navigator pane and select Refresh
If you just want the 'scriptedIteratorsPassingJavaObject' AL itself, then grab the _test/AssemblyLines/scriptedIteratorsPassingJavaObject.assemblyline file and drag-drop it into your own Project in the TDI CE. You can drag-drop most of the files found in the TDI Project (*.script, *.parser, *.attributemap, and so forth).
Now you should see the scriptedIteratorsPassingJavaObject AL there. Running it you should have both Iterators returning the same four Entries.
If you look in the Connection > Script for the first one you'll see it creates a HashMap and populates it with JS Objects. It then grabs the Op Entry from the AL (task) and saves the map as its '$map' attribute. This is all done in selectEntries().
In the second Iterator the selectEntries grabs the '$map' attribute from the AL's Op Entry and uses it for its own iteration.
Hope this helps! Please let me know if anything is unclear.
/Eddie
Hello Eddie.
Thank you a lot for this example and explanation!!!
It's work fine!
Thank you!
Sergey P.
Just an advice - when developing adapters it is important to ensure the non-functional attributes of your solution - so do not design the adapter that it will do things half it it breaks down in the middle of a process - also ensure if you are using things like hashmaps to store temporary data that you can actually trust the data of the map and that it can scale. This is mostly handled by TDI itself in most cases - but not thinking about these things in complex adapters (I assume this - simple adapters should not have a need of this kind) can bring you into troubles when the adapter is deployed and is met by the reality :-)

Good luck with your adapter.

Regards
Franz Wolfhagen
Sergey P
2019-08-11 08:02:07 UTC
Permalink
Post by Franzw
Post by Sergey P
Post by Eddie Hartman
Post by Sergey P
Hello guys
Excuse me for question that maybe has asked before, but I'm not found question described exactly like my case now.
I build adapter for IGI.
So, in short, I need to pass some Map object between two connectors that both defined in Feed on Iterator Mode. How can I do this?
I can also insert to one ScriptConnector some java code to writing data in file and then read from a file inside another ScriptConnector, but if I could to use more graceful way to implement this logic I'll glad to hear (read) :)
Thank you in advance!
Sergey P.
Ok, Sergey, as with most things TDI, there are several ways to flay this feline (aka skin the cat :)
One way is to pass stuff around using Input/Output Maps. However, your Iterators only have Input Maps for returning data. No easy way to pass in anything but string values via the parameters.
So instead, you could use the Op Entry which is owned by the AL. The Op Entry is created when the AL starts and it contains any instructions to the AL itself: settings, Connector parameters, AL Operation, etc. However, it is a valid Entry object and is around as long as the AL is running, so we can use it to pass around data within the AL.
I have created an example AL here: https://github.ibm.com/eddie-hartman/_test
1. Create a new Project named '_test'
2. Pull down the _test repo - either with 'git clone' or downloading the zip
Put this folder somewhere outside your TDI workspace.
3. Now copy the contents of this _test folder to the _test Project folder
in your TDI workspace.
4. Right-click on your _test Project in the TDI CE Navigator pane and select Refresh
If you just want the 'scriptedIteratorsPassingJavaObject' AL itself, then grab the _test/AssemblyLines/scriptedIteratorsPassingJavaObject.assemblyline file and drag-drop it into your own Project in the TDI CE. You can drag-drop most of the files found in the TDI Project (*.script, *.parser, *.attributemap, and so forth).
Now you should see the scriptedIteratorsPassingJavaObject AL there. Running it you should have both Iterators returning the same four Entries.
If you look in the Connection > Script for the first one you'll see it creates a HashMap and populates it with JS Objects. It then grabs the Op Entry from the AL (task) and saves the map as its '$map' attribute. This is all done in selectEntries().
In the second Iterator the selectEntries grabs the '$map' attribute from the AL's Op Entry and uses it for its own iteration.
Hope this helps! Please let me know if anything is unclear.
/Eddie
Hello Eddie.
Thank you a lot for this example and explanation!!!
It's work fine!
Thank you!
Sergey P.
Just an advice - when developing adapters it is important to ensure the non-functional attributes of your solution - so do not design the adapter that it will do things half it it breaks down in the middle of a process - also ensure if you are using things like hashmaps to store temporary data that you can actually trust the data of the map and that it can scale. This is mostly handled by TDI itself in most cases - but not thinking about these things in complex adapters (I assume this - simple adapters should not have a need of this kind) can bring you into troubles when the adapter is deployed and is met by the reality :-)
Good luck with your adapter.
Regards
Franz Wolfhagen
Hello Franz
Thank you for your observation.
I've been forced to use HashMap for Reconciliation process, when one ScriptConnector read data such as Account Attributes from HTTP response and save in HashMap Group ID as Key and Group Description as Value, and another ScriptConnector will be read data from Supporting Data and filling SD object class by Description pulled from HashMap for Group ID.
So, I think this way cover two point you mentioned - transient data and scalability.
Maybe this method not so graceful, but it work in adapter now :)
Anyway I'll be glad to hear for more efficiently method if such is exist.

Regards,
Sergey Paponov
Franzw
2019-08-12 06:11:12 UTC
Permalink
Post by Sergey P
Post by Franzw
Post by Sergey P
Post by Eddie Hartman
Post by Sergey P
Hello guys
Excuse me for question that maybe has asked before, but I'm not found question described exactly like my case now.
I build adapter for IGI.
So, in short, I need to pass some Map object between two connectors that both defined in Feed on Iterator Mode. How can I do this?
I can also insert to one ScriptConnector some java code to writing data in file and then read from a file inside another ScriptConnector, but if I could to use more graceful way to implement this logic I'll glad to hear (read) :)
Thank you in advance!
Sergey P.
Ok, Sergey, as with most things TDI, there are several ways to flay this feline (aka skin the cat :)
One way is to pass stuff around using Input/Output Maps. However, your Iterators only have Input Maps for returning data. No easy way to pass in anything but string values via the parameters.
So instead, you could use the Op Entry which is owned by the AL. The Op Entry is created when the AL starts and it contains any instructions to the AL itself: settings, Connector parameters, AL Operation, etc. However, it is a valid Entry object and is around as long as the AL is running, so we can use it to pass around data within the AL.
I have created an example AL here: https://github.ibm.com/eddie-hartman/_test
1. Create a new Project named '_test'
2. Pull down the _test repo - either with 'git clone' or downloading the zip
Put this folder somewhere outside your TDI workspace.
3. Now copy the contents of this _test folder to the _test Project folder
in your TDI workspace.
4. Right-click on your _test Project in the TDI CE Navigator pane and select Refresh
If you just want the 'scriptedIteratorsPassingJavaObject' AL itself, then grab the _test/AssemblyLines/scriptedIteratorsPassingJavaObject.assemblyline file and drag-drop it into your own Project in the TDI CE. You can drag-drop most of the files found in the TDI Project (*.script, *.parser, *.attributemap, and so forth).
Now you should see the scriptedIteratorsPassingJavaObject AL there. Running it you should have both Iterators returning the same four Entries.
If you look in the Connection > Script for the first one you'll see it creates a HashMap and populates it with JS Objects. It then grabs the Op Entry from the AL (task) and saves the map as its '$map' attribute. This is all done in selectEntries().
In the second Iterator the selectEntries grabs the '$map' attribute from the AL's Op Entry and uses it for its own iteration.
Hope this helps! Please let me know if anything is unclear.
/Eddie
Hello Eddie.
Thank you a lot for this example and explanation!!!
It's work fine!
Thank you!
Sergey P.
Just an advice - when developing adapters it is important to ensure the non-functional attributes of your solution - so do not design the adapter that it will do things half it it breaks down in the middle of a process - also ensure if you are using things like hashmaps to store temporary data that you can actually trust the data of the map and that it can scale. This is mostly handled by TDI itself in most cases - but not thinking about these things in complex adapters (I assume this - simple adapters should not have a need of this kind) can bring you into troubles when the adapter is deployed and is met by the reality :-)
Good luck with your adapter.
Regards
Franz Wolfhagen
Hello Franz
Thank you for your observation.
I've been forced to use HashMap for Reconciliation process, when one ScriptConnector read data such as Account Attributes from HTTP response and save in HashMap Group ID as Key and Group Description as Value, and another ScriptConnector will be read data from Supporting Data and filling SD object class by Description pulled from HashMap for Group ID.
So, I think this way cover two point you mentioned - transient data and scalability.
Maybe this method not so graceful, but it work in adapter now :)
Anyway I'll be glad to hear for more efficiently method if such is exist.
Regards,
Sergey Paponov
I believe what you describe here is exactly when you can use transient storage objects in the process - if the process fails in the middle it should be cleaned up under all circumstances.

Whether a hashmap or a more simple object to hold the groups is the optimal depends on details and coding preferences - hashmaps are good because you can look things up whereas e.g. arrays or lists needs to processed in sequence or through an index which may cause more convoluted logic if you are looking for something specific :-)

Normally a WS based system adapter can be solved by wrapping the WS reconciliation in a scripted connector. I have been playing with the idea of creating a connector that can wrap another connector to make this easier - and this seems IMHO to be a pretty good idea - but my time is limited so I have not done this yet - feel free to take this idea :-)

And good to see you coming back - keep posting and if you run into other problems let the community know so we can discuss and help out :-)

Regards
Franz Wolfhagen
Eddie Hartman
2019-08-12 17:15:44 UTC
Permalink
Post by Franzw
Post by Sergey P
Post by Franzw
Post by Sergey P
Post by Eddie Hartman
Post by Sergey P
Hello guys
Excuse me for question that maybe has asked before, but I'm not found question described exactly like my case now.
I build adapter for IGI.
So, in short, I need to pass some Map object between two connectors that both defined in Feed on Iterator Mode. How can I do this?
I can also insert to one ScriptConnector some java code to writing data in file and then read from a file inside another ScriptConnector, but if I could to use more graceful way to implement this logic I'll glad to hear (read) :)
Thank you in advance!
Sergey P.
Ok, Sergey, as with most things TDI, there are several ways to flay this feline (aka skin the cat :)
One way is to pass stuff around using Input/Output Maps. However, your Iterators only have Input Maps for returning data. No easy way to pass in anything but string values via the parameters.
So instead, you could use the Op Entry which is owned by the AL. The Op Entry is created when the AL starts and it contains any instructions to the AL itself: settings, Connector parameters, AL Operation, etc. However, it is a valid Entry object and is around as long as the AL is running, so we can use it to pass around data within the AL.
I have created an example AL here: https://github.ibm.com/eddie-hartman/_test
1. Create a new Project named '_test'
2. Pull down the _test repo - either with 'git clone' or downloading the zip
Put this folder somewhere outside your TDI workspace.
3. Now copy the contents of this _test folder to the _test Project folder
in your TDI workspace.
4. Right-click on your _test Project in the TDI CE Navigator pane and select Refresh
If you just want the 'scriptedIteratorsPassingJavaObject' AL itself, then grab the _test/AssemblyLines/scriptedIteratorsPassingJavaObject.assemblyline file and drag-drop it into your own Project in the TDI CE. You can drag-drop most of the files found in the TDI Project (*.script, *.parser, *.attributemap, and so forth).
Now you should see the scriptedIteratorsPassingJavaObject AL there. Running it you should have both Iterators returning the same four Entries.
If you look in the Connection > Script for the first one you'll see it creates a HashMap and populates it with JS Objects. It then grabs the Op Entry from the AL (task) and saves the map as its '$map' attribute. This is all done in selectEntries().
In the second Iterator the selectEntries grabs the '$map' attribute from the AL's Op Entry and uses it for its own iteration.
Hope this helps! Please let me know if anything is unclear.
/Eddie
Hello Eddie.
Thank you a lot for this example and explanation!!!
It's work fine!
Thank you!
Sergey P.
Just an advice - when developing adapters it is important to ensure the non-functional attributes of your solution - so do not design the adapter that it will do things half it it breaks down in the middle of a process - also ensure if you are using things like hashmaps to store temporary data that you can actually trust the data of the map and that it can scale. This is mostly handled by TDI itself in most cases - but not thinking about these things in complex adapters (I assume this - simple adapters should not have a need of this kind) can bring you into troubles when the adapter is deployed and is met by the reality :-)
Good luck with your adapter.
Regards
Franz Wolfhagen
Hello Franz
Thank you for your observation.
I've been forced to use HashMap for Reconciliation process, when one ScriptConnector read data such as Account Attributes from HTTP response and save in HashMap Group ID as Key and Group Description as Value, and another ScriptConnector will be read data from Supporting Data and filling SD object class by Description pulled from HashMap for Group ID.
So, I think this way cover two point you mentioned - transient data and scalability.
Maybe this method not so graceful, but it work in adapter now :)
Anyway I'll be glad to hear for more efficiently method if such is exist.
Regards,
Sergey Paponov
I believe what you describe here is exactly when you can use transient storage objects in the process - if the process fails in the middle it should be cleaned up under all circumstances.
Whether a hashmap or a more simple object to hold the groups is the optimal depends on details and coding preferences - hashmaps are good because you can look things up whereas e.g. arrays or lists needs to processed in sequence or through an index which may cause more convoluted logic if you are looking for something specific :-)
Normally a WS based system adapter can be solved by wrapping the WS reconciliation in a scripted connector. I have been playing with the idea of creating a connector that can wrap another connector to make this easier - and this seems IMHO to be a pretty good idea - but my time is limited so I have not done this yet - feel free to take this idea :-)
And good to see you coming back - keep posting and if you run into other problems let the community know so we can discuss and help out :-)
Regards
Franz Wolfhagen
That's an excellent idea, Franz. If you (and anyone else who wants to get online) would like to have a look at this, I'm game :D As long as someone has access to a service against we can test as we interatively build the component.
n***@gmail.com
2019-08-15 01:31:39 UTC
Permalink
hey Eddie ... could you post a link to the assembly line ... thx
Franzw
2019-08-16 08:12:02 UTC
Permalink
Post by n***@gmail.com
hey Eddie ... could you post a link to the assembly line ... thx
I think you are somewhat overly optimistic here - I do not see the mention of any build ALs in the conversation here - only instructions and ideas...

Regards
Franz Wolfhagen

Loading...