Running Transformations

Once created, Transform4J makes transformations easy to run. Set up your source and target and then run it. An example can be found below. Utility class Transform4j has been provided to make running transformations easy. See javadoc.

JdbcDataSource source = new JdbcDataSource(myDataSource, "select * from customer");
CsvDataTarget target = new CsvDataTarget(new File("c:/extracts/myFile.csv"));

MyTransformation myTransformation = new MyTransformation(source, target);
Transform4j.runTransformation(myTransformation);
			
Alternatively, you can embed your transformation in a preconfigured Runnable that can be used with products like Spring Batch or Quartz.
Runnable transformationRunnable = Transform4j.makeRunnable(myTransformation);
			

Scaling Up -- Running groups of transformations in parallel

The Transform4J world incents developers toward smaller, less complex transformations. As a consequence in the real world, developers will have multiple transformations used in combination or in sequence. For example, if you're copying customer information from one application database to another, there may be many related constructs in the copy. For instance, each customer might have multiple addresses, phone numbers, email addresses, credit cards, and preferences. You'll likely need to copy direct customer information (e.g. name, some sort of identifier) before copying addresses, phone numbers email addresses, etc. Furthermore, some of these constructs (e.g. addresses) rely on other constructs, such as state and country. In a Transform4J world, this will likely need to be broken out in several different transformations and some of them will need to run before others. Transform4J supports running sets of related transactions through the TransformerExecutionGroup

Transformer execution groups are implement Runnable and Callable so they can easily be executed directly or through batch scheduling products. An example of setup and execution follows. In this example, all transformers will be run concurrently.

TransformerExecutionGroup executionGroup = Transform4j.makeExecutionGroup(transform1a, transform1b, transform1c, transform1d);
executionGroup.setMaxNbrThreads(4);
executionGroup.run();
				

As you see from the example above, the maximum number of transformations run concurrently is configurable. By default, that number is Runtime.getRuntime().availableProcessors() * 3. All transformers must be preconfigured with sources and targets.

With a set of logically related transformations, it's logical to expect that some of those transformations can run concurrently (e.g. in parallel) and some transformations will depend on the output of other transformations. Transformer execution groups support this need by allowing entire transformer groups to be chained. That is, it's easy to line up a chain of execution groups that will be run one after the other in consecutive fashion. An example of this follows.

In this example, transformers 1a through 1d will be run to completion. Then transformers 2a through 2f will be run to completion. Then transformers 3a and 3b will run. At most, four transformers will run at one time.

TransformerExecutionGroup executionGroup = Transform4j.makeExecutionGroup(transform1a, transform1b, transform1c, transform1d);

TransformerExecutionGroup executionGroup2 = Transform4j.makeExecutionGroup(transform2a, transform2b, transform2c, 
				transform2d, transform2e, transform2f);
executionGroup.setNextTransformerExecutionGroup(executionGroup2);

TransformerExecutionGroup executionGroup3 = Transform4j.makeExecutionGroup(transform3a, transform3b);
executionGroup2.setNextTransformerExecutionGroup(executionGroup3);
		
executionGroup.setMaxNbrThreads(4);
executionGroup.run();
				

Note that I only specified the maximum number of threads on the main execution group that I ran. Execution groups 2 and 3 will also honor the four thread limitation, unless I explicitly specify another thread limit for them.

Next section.