Discussion:
Fail iteration without failing entire assembly line run
(too old to reply)
Scott Ellis
2017-06-06 18:35:15 UTC
Permalink
Raw Message
I have some assembly lines in production that are reporting errors manually instead of throwing exceptions. I do this because I don't want to drop all the way out of an assembly line due to an error that occurs in processing one row of an iteration.

Here is some sample code that demonstrates the issue in a simple ScriptConnector (TDI 7.1.1, btw):

var counter = 0;
var indexesToFailOn = [10, 20, 30, 70, 95];

function selectEntries() {
counter = 0;
}

function failIterationWithoutKillingAssemblyLine() {
throw new java.lang.IllegalArgumentException(
"Intentional failure thrown on index " + indexesToFailOn[counter]);
}

function getNextEntry () {
if (counter > 100) {
result.setStatus (0);
result.setMessage ("End of input");
return;
}

if (counter in indexesToFailOn) {
failIterationWithoutKillingAssemblyLine();
}

entry.setAttribute ("counter", counter);
counter++;
}

So as you can see, this connector is assigned to an assembly line and runs for 101 iterations, failing selectively on indexes from the indexesToFailOn array. My runtime output is this:

11:17:06,574 INFO - CTGDIS087I Iterating.
11:17:06,574 ERROR - [Fail Iteration] CTGDIS810E handleException - cannot handle exception , get
11:17:06,574 ERROR - CTGDIS266E Error in NextIteratorEntry. Exception occurred: java.lang.IllegalArgumentException: Intentional failure thrown on index 10
11:17:06,574 INFO - CTGDIS100I Printing the Connector statistics.
11:17:06,574 INFO - [Fail Iteration] Errors:1
11:17:06,574 INFO - CTGDIS104I Total: Errors:2.
11:17:06,574 INFO - CTGDIS101I Finished printing the Connector statistics.
11:17:06,574 ERROR - CTGDIS077I Failed with error: Intentional failure thrown on index 10.

I read an article about error hooks, which would prevent the thrown exception from kicking out of the whole assembly line run. However, at the end of the run, I want an accurate accounting of the number of rows had errors. Even in this simple example its reporting two errors when there was only one.

Here is the output of the above if the indexesToFailOn array is empty ([]):

11:24:13,072 INFO - CTGDIS087I Iterating.
11:24:13,072 INFO - CTGDIS088I Finished iterating.
11:24:13,072 INFO - CTGDIS100I Printing the Connector statistics.
11:24:13,072 INFO - [Fail Iteration] Get:101
11:24:13,072 INFO - CTGDIS104I Total: Get:101.
11:24:13,072 INFO - CTGDIS101I Finished printing the Connector statistics.
11:24:13,072 INFO - CTGDIS080I Terminated successfully (0 errors).

In my real implementation, I'm detecting errors, either wrapping and catching exceptions or checking for invalid situations, then reporting it via the task.logmsg method. With that strategy, it appears at the end just like the output above:

11:24:13,072 INFO - CTGDIS080I Terminated successfully (0 errors).


What I'm looking for is this:

- How can my failIterationWithoutKillingAssemblyLine function above be changed, or specifically what kind of error hook or other mechanism be enabled, that will report the error like a real error, be counted only once in the final tally (instead of twice as in this case), and show up at the end of the assembly line run with the correct number of errors? In this case the last line would look something like:

11:24:13,072 INFO - CTGDIS080I Terminated 101 processed, 96 successful, 5 errors

Something like that. The point is, I want to see all the errors but I want to process the entire assembly line.
Eddie Hartman
2017-06-07 19:23:39 UTC
Permalink
Raw Message
Post by Scott Ellis
I have some assembly lines in production that are reporting errors manually instead of throwing exceptions. I do this because I don't want to drop all the way out of an assembly line due to an error that occurs in processing one row of an iteration.
var counter = 0;
var indexesToFailOn = [10, 20, 30, 70, 95];
function selectEntries() {
counter = 0;
}
function failIterationWithoutKillingAssemblyLine() {
throw new java.lang.IllegalArgumentException(
"Intentional failure thrown on index " + indexesToFailOn[counter]);
}
function getNextEntry () {
if (counter > 100) {
result.setStatus (0);
result.setMessage ("End of input");
return;
}
if (counter in indexesToFailOn) {
failIterationWithoutKillingAssemblyLine();
}
entry.setAttribute ("counter", counter);
counter++;
}
11:17:06,574 INFO - CTGDIS087I Iterating.
11:17:06,574 ERROR - [Fail Iteration] CTGDIS810E handleException - cannot handle exception , get
11:17:06,574 ERROR - CTGDIS266E Error in NextIteratorEntry. Exception occurred: java.lang.IllegalArgumentException: Intentional failure thrown on index 10
11:17:06,574 INFO - CTGDIS100I Printing the Connector statistics.
11:17:06,574 INFO - [Fail Iteration] Errors:1
11:17:06,574 INFO - CTGDIS104I Total: Errors:2.
11:17:06,574 INFO - CTGDIS101I Finished printing the Connector statistics.
11:17:06,574 ERROR - CTGDIS077I Failed with error: Intentional failure thrown on index 10.
I read an article about error hooks, which would prevent the thrown exception from kicking out of the whole assembly line run. However, at the end of the run, I want an accurate accounting of the number of rows had errors. Even in this simple example its reporting two errors when there was only one.
11:24:13,072 INFO - CTGDIS087I Iterating.
11:24:13,072 INFO - CTGDIS088I Finished iterating.
11:24:13,072 INFO - CTGDIS100I Printing the Connector statistics.
11:24:13,072 INFO - [Fail Iteration] Get:101
11:24:13,072 INFO - CTGDIS104I Total: Get:101.
11:24:13,072 INFO - CTGDIS101I Finished printing the Connector statistics.
11:24:13,072 INFO - CTGDIS080I Terminated successfully (0 errors).
11:24:13,072 INFO - CTGDIS080I Terminated successfully (0 errors).
11:24:13,072 INFO - CTGDIS080I Terminated 101 processed, 96 successful, 5 errors
Something like that. The point is, I want to see all the errors but I want to process the entire assembly line.
I would do something like this:

1. In the Options > AssemblyLine Hooks put this into Prolog - Before Init:
errors = {} // create an empty object to store errors

2. Under Resources > Scripts you create 'default_OnError' with:
name = thisConnector.getName(); // get Connector name
task.logmsg("ERROR", "Error for " + name + " - " + error.toJSON());
errors[name] = (errors[name] || 0) + 1; // +1 for this Connector, initializing counter to 0 on first time

3. Drag the above default_OnError Script onto the Default On Error Hook of all Connectors you want to catch errors for.

4. In the Options > AssemblyLine Hooks drop this into the On Success Hook.
total = 0;
for (name in errors) {
task.logmsg("Errors for " + name + ": " + errors[name]);
total++;
}
task.logmsg("Total errors: " + total);

?. If you also want the above code for On Error, create a Script under Resources > Scripts and drag it onto both. Or you can drop the script into the Eplilog or Eplilog - After Close, both of which are always executed on AL termination.
Scott Ellis
2017-06-07 21:36:11 UTC
Permalink
Raw Message
Thanks for the reploy, Eddie--I'll be giving this a go tomorrow morning sometime and will let you know how it works out.
j***@gmail.com
2017-06-08 15:41:25 UTC
Permalink
Raw Message
Ok, great news--I have exactly what I want. Here is the final solution, drawing from everything you have outlined above:

I was able to do everything in the script connector, and didn't need to do anything special to the hooks of the assembly line itself. By providing the connector with a DataFlow (Iterator) -> Default On Error hook, and another for its Prolog -> Before Initialize script, I get the following output:

08:32:25,202 INFO - CTGDIS087I Iterating.
08:32:25,202 ERROR - Error for Fail Selected Iterations - Intentional failure thrown on index 10
08:32:25,202 ERROR - Error for Fail Selected Iterations - Intentional failure thrown on index 20
08:32:25,202 ERROR - Error for Fail Selected Iterations - Intentional failure thrown on index 30
08:32:25,202 ERROR - Error for Fail Selected Iterations - Intentional failure thrown on index 70
08:32:25,202 ERROR - Error for Fail Selected Iterations - Intentional failure thrown on index 95
08:32:25,202 INFO - CTGDIS088I Finished iterating.
08:32:25,202 INFO - CTGDIS100I Printing the Connector statistics.
08:32:25,202 INFO - [Fail Selected Iterations] Get:96, Errors:5
08:32:25,202 INFO - CTGDIS104I Total: Get:96, Errors:5.
08:32:25,202 INFO - CTGDIS101I Finished printing the Connector statistics.
08:32:25,202 INFO - CTGDIS080I Terminated successfully (5 errors).

The final output works for me without providing anything with the on Success Assembly Line Hook.

Here are the steps I took:

1. Create a Script Connector called Fail Selected Iterations, Iterator mode.
2. Edit the connector, tab Hooks, set the Prolog -> Before Initialize script to:

// Create an empty object to store errors
errors = {};

3. Edit the connector, tab Hooks, set the DataFlow (Iterator) -> Default On Error script and set to (could also create the script in Scripts as you described and drag and drop it onto the same hook):

name = thisConnector.getName();
task.logmsg("ERROR", "Error for " + name + " - " + error.message);

// +1 for this connector, initializing to 0 on first time
errors[name] = (errors[name] || 0) + 1;

4. Edit the the connector, Connection tab, Edit Script button, set script to:

var counter = 0;
var indexesToFailOn = [10, 20, 30, 70, 95];

function selectEntries() {
counter = 0;
}

function failIterationWithoutKillingAssemblyLine(index) {
throw new java.lang.IllegalArgumentException(
"Intentional failure thrown on index " + index);
}

function isFailureIndex(index) {
for (var i=0; i < indexesToFailOn.length; i++) {
if (index == indexesToFailOn[i]) {
return true;
}
}

return false;
}

function getNextEntry () {
var entryNum = counter++;

if (entryNum > 100) {
result.setStatus (0);
result.setMessage ("End of input");
return;
}

if (isFailureIndex(entryNum)) {
failIterationWithoutKillingAssemblyLine(entryNum);
}
}

5. Create an assembly line and add this connector to its Feed.

6. Run the assembly line and see the output as shown above.

Voila!

Thank you.
j***@gmail.com
2017-06-08 16:26:17 UTC
Permalink
Raw Message
Ah, so now I have a slight variation of the same problem, but this time, for every iteration of my connector, I need any exception thrown by any of the script connectors in the data flow to count as a failed iteration and have that failure cause the row to be skipped but counted as 1 error.

I've modified my test connector to set an errorMessage instead of throwing out:

if (isFailureIndex(entryNum)) {
entry["errorMessage"] = "Intentional failure thrown on index " + entryNum;
// failIterationWithoutKillingAssemblyLine(entryNum);
} else {
entry["errorMessage"] = "";
}

I added a script connector and added to the assembly lines data flow:

if (work.errorMessage && work.errorMessage[0].length) {
throw new java.lang.IllegalArgumentException(work.errorMessage[0]);
}

I throw out of the entire assembly line as before now, so the exception handler is outside the purview of the feed connector and doesn't get caught. I'm going to try a few things but I'm wondering if there is a prescribed way to handle this situation?
Eddie Hartman
2017-06-09 09:55:21 UTC
Permalink
Raw Message
Post by j***@gmail.com
Ah, so now I have a slight variation of the same problem, but this time, for every iteration of my connector, I need any exception thrown by any of the script connectors in the data flow to count as a failed iteration and have that failure cause the row to be skipped but counted as 1 error.
if (isFailureIndex(entryNum)) {
entry["errorMessage"] = "Intentional failure thrown on index " + entryNum;
// failIterationWithoutKillingAssemblyLine(entryNum);
} else {
entry["errorMessage"] = "";
}
if (work.errorMessage && work.errorMessage[0].length) {
throw new java.lang.IllegalArgumentException(work.errorMessage[0]);
}
I throw out of the entire assembly line as before now, so the exception handler is outside the purview of the feed connector and doesn't get caught. I'm going to try a few things but I'm wondering if there is a prescribed way to handle this situation?
Once again the Error Hooks could be used here to send processing back to the Feed section for another iteration: system.skipEntry();

Note that skipEntry() requires you to have an iterator in the Feed section. Another option is to use system.exitFlow() which simply prevents the rest of the Flow components for doing anything, and if you have a Feed Iterator then control is passed back to it.
j***@gmail.com
2017-06-09 18:23:51 UTC
Permalink
Raw Message
Ok, that sounds promising. I always have an iterator in my feed section so that part is of no concern to me. If I skip the entry either by system.exitFlow or system.skipEntry, will that add to the internal tally of errors? I need that number to increment by one for every error I encounter. I have some code that comes just short of allowing me to set the number of errors explicitly. At the end of processing, I look to that map of errors you had me create to see how many errors there were, if any. Then I set the stats into the task:

var statsEntry = new com.ibm.di.entry.Entry();
statsEntry["err"] = numErrors;
task.getStats().setStats(statsEntry);

This works to set the errors, but it blows away the other counts. If I try to set statsEntry["get"] to anything, an exception is thrown. I'm really just fumbling around in the dark trying to trick this into working, as you might have guessed.
j***@gmail.com
2017-06-09 19:07:29 UTC
Permalink
Raw Message
Hi, Eddie,

I tried out what you've suggested and it seems to work great. I converted my scripts in the data flow section to be script connectors in AddOnly mode--I put the script in putEntry() that I was previously calling as a plain script.

The very interesting thing is that I no longer need any hooks in my iterator (no need to initialize an error map or trap errors on the default on error hook).

The new script connector in AddOnly mode has one method, putEntry():

function putEntry () {
var errorMessage = task.getCurrentWork()["errorMessage"];
if (errorMessage && errorMessage[0].length) {
var assemblyLineName = task.getShortName();
system.throwException(errorMessage);
}
}

The errorMessage work attribute is set by the iterator to trigger this code periodically during iteration.

I've overridden the script connector's Default On Error hook and added this:

task.logmsg("ERROR", error.message);
system.exitFlow();

And the output magically works just as I want it to:

12:05:59,542 INFO - CTGDIS087I Iterating.
12:05:59,542 ERROR - Intentional failure thrown on index 1
12:05:59,542 ERROR - Intentional failure thrown on index 20
12:05:59,559 ERROR - Intentional failure thrown on index 30
12:05:59,559 ERROR - Intentional failure thrown on index 70
12:05:59,559 ERROR - Intentional failure thrown on index 95
12:05:59,559 INFO - CTGDIS088I Finished iterating.
12:05:59,559 INFO - CTGDIS100I Printing the Connector statistics.
12:05:59,559 INFO - [Fail Selected Iterations] Get:100
12:05:59,559 INFO - [Throw Exception If Found] Add:95, Errors:5
12:05:59,559 INFO - CTGDIS104I Total: Get:100, Add:95, Errors:5.
12:05:59,559 INFO - CTGDIS101I Finished printing the Connector statistics.
12:05:59,559 INFO - CTGDIS080I Terminated successfully (5 errors).

Now hopefully I haven't left out some other nuances.

Thank you!
Eddie Hartman
2017-06-10 17:34:10 UTC
Permalink
Raw Message
Post by j***@gmail.com
Hi, Eddie,
I tried out what you've suggested and it seems to work great. I converted my scripts in the data flow section to be script connectors in AddOnly mode--I put the script in putEntry() that I was previously calling as a plain script.
The very interesting thing is that I no longer need any hooks in my iterator (no need to initialize an error map or trap errors on the default on error hook).
function putEntry () {
var errorMessage = task.getCurrentWork()["errorMessage"];
if (errorMessage && errorMessage[0].length) {
var assemblyLineName = task.getShortName();
system.throwException(errorMessage);
}
}
The errorMessage work attribute is set by the iterator to trigger this code periodically during iteration.
task.logmsg("ERROR", error.message);
system.exitFlow();
12:05:59,542 INFO - CTGDIS087I Iterating.
12:05:59,542 ERROR - Intentional failure thrown on index 1
12:05:59,542 ERROR - Intentional failure thrown on index 20
12:05:59,559 ERROR - Intentional failure thrown on index 30
12:05:59,559 ERROR - Intentional failure thrown on index 70
12:05:59,559 ERROR - Intentional failure thrown on index 95
12:05:59,559 INFO - CTGDIS088I Finished iterating.
12:05:59,559 INFO - CTGDIS100I Printing the Connector statistics.
12:05:59,559 INFO - [Fail Selected Iterations] Get:100
12:05:59,559 INFO - [Throw Exception If Found] Add:95, Errors:5
12:05:59,559 INFO - CTGDIS104I Total: Get:100, Add:95, Errors:5.
12:05:59,559 INFO - CTGDIS101I Finished printing the Connector statistics.
12:05:59,559 INFO - CTGDIS080I Terminated successfully (5 errors).
Now hopefully I haven't left out some other nuances.
Thank you!
Nice, Andrew. Remember, you can also pass Javascript objects as Attribute variables. So instead of just making the errors object, you can assign it into work to carry along the AL. Or even better, to the opEntry, which is not zeroed out at the top of each loop.

--- AL Prolog Before Init (or) Connector Prolog Before Init:
opEntry = task.getOpEntry();
opEntry.errors = {};
---

I'm a big fan of JS objects you can see :)

Loading...