Writing your own targets

This section provides instructions on how to develop your own Transform4J Data Targets.

Data targets implement interface TransformerDataTarget, which provides a way for Transformer classes to insert data into the target. Optionally, targets can also implement the Updatable interface that adds the ability to saveOrUpdate, update, or delete. Let's start with a simpler example of implementing a target that can only insert data; we'll look at a more complicated example that can update and delete data as well later.

Topics on this page:

Target responsibilities

This section outlines basic target responsibilities.

All data targets implement interface TransformerDataTarget. See TransformerDataTarget for more detail. All targets also implement interface Insertable, which provides basic insert/append capability. Targets that have the ability to do keyed lookups (e.g. relational and NoSql databases) can optionally implement the Updatable interface, which provides the additional ability to perform keyed updates and deletes.

Targets are responsible for managing resources needed for I/O. This means closing any resources they open to write data. For instance, the JdbcDataTarget class is responsible for closing any Statement or Connection resources it opens. The CsvDataTarget and JsonDataTarget classes are responsible for closing file output streams they open write their output.

Targets are responsible for reporting I/O activity (e.g. statistics for all writes). These statistics are available to users running transformations and can be useful in verifying that transformations accomplished what was wanted.

Targets are responsible for providing trigger support. All targets must support insert triggers, which allow users to have code invoked before and after insert operations.. All insert triggers implement interface InsertTrigger. Updatable targets must also support delete and update triggers, which allow users to have code invoked before and after update and delete operations. Update and delete triggers implment the UpdateTrigger and DeleteTrigger interfaces respectively.

As these requirements for data targets are lengthy, Transform4J provides base class assistance. Data targets that extend BaseDataTarget automatically satisfy the activity reporting and trigger support requirements. Updatable targets should extend BaseUpdatableDataTarget instead as it also provides activity reporting and trigger support for update and delete operations.

Important Javadoc references for targets are as follows:

  • TransformerDataTarget
  • Insertable
  • Updatable
  • InsertTrigger
  • UpdateTrigger
  • DeleteTrigger
  • Writing targets that inserts only

    All data targets support basic data writes or "inserts". I'm going to leverage JsonDataTarget.

    Step 1: Create class that extends BaseDataTarget. You'll be forced to implement several methods. While you are not required to override init(), most targets do as they have specific resources to close. If you do override init(), be sure to call super.init() or the target won't work.

    public class JsonDataTarget extends BaseDataTarget<DataRecord> {
    
    	@Override
    	public void close() {
    		// TODO Close all resources here
    	}
    	
    	@Override
    	protected void localInsert(DataRecord record) {
    		// TODO Put your 'insert' logic here
    	}
    	
    	@Override
    	public void init() {
    		super.init();
    		
    		// Put any resource initialization here
    	}
    }
    				

    Step 2: Validate any additional inputs your target requires. All validation should be executed when init() is called. The Json target only requires an OutputStream, however, users can supply an output file (which can easily be turned into an OutputStream), in which case that file must be writable. An example of that validation follows:

    public class JsonDataTarget extends BaseDataTarget<DataRecord> {
    
    	@Override
    	public void init() {
    		super.init();
    		
    		if (this.getOutputFile() != null) {
            	Validate.notNull(this.getOutputFile(), "Null outputFile not allowed.");
            	Validate.isTrue(this.getOutputFile().canWrite(), "File not writable.  file=" 
            			+ this.getOutputFile().getAbsolutePath());
            	Validate.isTrue(!this.getOutputFile().isDirectory(), "File is a directory -- must specify file name.  file=" 
            			+ this.getOutputFile().getAbsolutePath());
            	try {
        			this.setOutStream(new FileOutputStream(this.getOutputFile()));
        		} catch (FileNotFoundException e) {
        			throw new Transform4jRuntimeException(e)
        				.addContextValue("file", this.getOutputFile().getAbsolutePath());
        		}
            	
            	if (StringUtils.isEmpty(this.getOutputLabel())) {
            		this.setOutputLabel(this.getOutputFile().getName());
            	}
            }
            
            Validate.notEmpty(this.getOutputLabel(), "Null or blank outputLabel not allowed");
            
            // Additional code omitted for brevity.
    	}
    	
    }
    				

    Under the covers, this target relies on Google Gson for producing Json output. Gson requires we setup a JsonWriter. An example follows:

    public class JsonDataTarget extends BaseDataTarget<DataRecord> {
    
    	private JsonWriter writer;
    	private Gson gson;
    	
    	@Override
    	public void init() {
    		// Additional code omitted for brevity.
    		
    		try {
    			writer = new JsonWriter(new OutputStreamWriter(outStream, "UTF-8"));
    			writer.setIndent("  ");
    			writer.beginArray();
    		} catch (Exception e) {
    			throw new Transform4jRuntimeException("Error opening Json writer",e);
    		} 
    		
    		GsonBuilder builder = new GsonBuilder();
    		builder.registerTypeHierarchyAdapter(DataRecord.class, new DataRecordGsonSerializer());
    		if (prettyPrinting) {
    			builder.setPrettyPrinting();
    		}
    		gson = builder.create();
    	}
    }
    				

    Step 3: Provide insert logic. For this target, this is relatively simple. An example follows:

    public class JsonDataTarget extends BaseDataTarget<DataRecord> {
    
    	@Override
    	protected void localInsert(DataRecord record) {
    		Validate.isTrue(initCalled,"init() not called.");
    		Validate.isTrue(record.isKeysCaseSensitive(), "DataRecord *must* be case sensitive for JSON");
    		gson.toJson(record, DataRecord.class, writer);
    	}
    }
    				

    Step 4: Close all resources. For this target, the writer needs to be closed, but that's it.

    public class JsonDataTarget extends BaseDataTarget<DataRecord> {
    
    	@Override
    	public void close() {
    		try {
    			writer.endArray();
    			writer.close();
    		} catch (Exception e) {
    			throw new Transform4jRuntimeException("Error closing Json writer",e);
    		}
    	}
    }
    				

    That's it -- take your new target out for a test drive!

    Writing targets that also support update and delete operations

    Updatable targets are required to provide everything listed above. In addition, they provide logic to perform updates, deletes, and keyed lookups. I'm going to use target MongoDbDataTarget as an example.

    Step 1: Create class that extends BaseUpdatableDataTarget. You'll be required to implement all the same methods as in the previous example. In addition, you'll be required to implement several additional methods:

    public class MongoDbDataTarget extends BaseUpdatableDataTarget<DataRecord> {
    
    	@Override
    	public void saveOrUpdate(DataRecord record) {
    		// TODO put save or update logic here
    	}
    	
    	@Override
    	protected int localUpdate(DataRecord record) {
    		// TODO put keyed update logic here
    	}
    	
    	@Override
    	protected int localDelete(DataRecord record) {
    		// TODO put keyed delete logic here.
    	}
    }
    				

    Perform steps 2, 3, and 4 from above.

    Step 5: Implement saveOrUpdate() logic. This method does a keyed read. If data is found using the key values provided in the input, then an update is performed. Otherwise, an insert is performed. An example follows. Note that the key fields needed for the lookup are at an instance level and you have access to them.

    public class MongoDbDataTarget extends BaseUpdatableDataTarget<DataRecord> {
    
    	@Override
    	public void saveOrUpdate(DataRecord record) {
    		DBObject query = this.createFieldList(this.getKeyFields(), record);
    		BasicDBObject fields = new BasicDBObject("_id",1);
    		DBCursor mongoCursor = null;
    		try{
    			mongoCursor = mongoDbCollection.find(query);
    			if (mongoCursor.hasNext()) {
    				this.update(record);
    			}
    			else {
    				this.insert(record);
    			}
    		}
    		finally {
    			MongoDbUtils.closeQuietly(mongoCursor);
    		}	
    	}
    }
    				

    Step 5: Implement keyed update logic. Note that implementors are required to return the number of items affected by the update.

    public class MongoDbDataTarget extends BaseUpdatableDataTarget<DataRecord> {
    
    	@Override
    	protected int localUpdate(DataRecord record) {
    		DBObject query = this.createFieldList(this.getKeyFields(), record);
    		DBObject update = new BasicDBObject("$set", dbObjectConverter.convert(record));
    		WriteResult result = mongoDbCollection.updateMulti(query, update);
    		return result.getN();
    	}			
    }
    				

    Step 6: Implement keyed delete logic. Note that implementors are required to return the number of items affected by the delete.

    public class MongoDbDataTarget extends BaseUpdatableDataTarget<DataRecord> {
    		
    	@Override
    	protected int localDelete(DataRecord record) {
    		DBObject query = this.createFieldList(this.getKeyFields(), record);
    		WriteResult result = mongoDbCollection.remove(query);
    		return result.getN();
    	}		
    }
    				

    That's it -- take your new target out for a test drive!