From UbikWiki

Current revision

Development of a batch

Type of batches

Batches

Batches are dedicated to treatments that have to work with a big volume of datas. Batches use full performance features of the server:

• Integrated multi-threading
• Handling of large sets of data
• Opportunity to interrupt a batch when the method Traitement#exec(Object[] datas) returns
• Ability to modify parameters of Query from Web Controler

Tasks

Tasks are useful when the volume of data is not important. Using tasks, your treatment will be mono threaded.

Here are the characteristics of a batch :

1. You define a starting request. This request will retrieve your basic datas. You can define some parameters which will replace bind variables in the starting request.
2. Console batch will share the result of the starting between threads, thus all datas are not loaded in memory, but only what has been allocated to the threads.
3. Each Thread works simultaneously, carrying out the same treatment, but with datas allocated to it, thus your increase the performance of your treatment.
4. You can link several batches themselves. By example, once a batch has finished, it can start an other one.

Methods of a batch

Your batch class has to extend class:

• After 5.0.0.0 : com.ubikingenierie.console.apiconsole.BaseBatch
• Before 5.0.0.0 : salto.batch.lanceur.thread.Traitement.

Two methods must be implemented :

• executeOnBlock() (exec() before 5.0.0.0): treatment executed by each thread
• createConfiguration() (public void static registerAsBatch() before 5.0.0.0): creates the Batch configuration (ie the parameters of the batch (number of threads, definition of the starting request...))

Two others methods can be implemented :

• executeBeforeData() (initialize() before 5.0.0.0)
• executeAfterDataEnd() (finalizeBatch() before 5.0.0.0)

initialize and finalizeBatch are executed in a mono threaded context.

BaseBatch#executeBeforeData() (Traitement#initialize() before 5.0.0.0)

The initialize method is the first method executed at the start of your batch and before IBatchStart#init() is called. This means you can modify the SQL Query parameters in this method. Insert in this method all the code you want to be executed before getting the Data from Database. It is generally some initializing treatments.

executeOnBlock(Object[] data) (Traitement#exec(Object[] data) before 5.0.0.0)

The exec method is the method executed by each thread. In this method, you will call your business code. The Batch Server will pass as parameters the Data that it retrieved from database.

BaseBatch#executeAfterDataEnd() (Traitement#finalizeBatch() before 5.0.0.0)

The code implemented in this method will be executed once all threads have finished to treat all datas from the starting request or after an error has occured and the batch is in STOP_ON_ERROR mode.

BaseBatch#createConfiguration() (public static Traitement#registerAsBatch() before 5.0.0.0)

The registerAsBatch method is executed when a BAR is deployed. This method holds the configuration of a Batch:

• IBatch Start implementation to use and its parameters
• IBatchAppender implementation to use and its parameters
• AbstractLinker(s) subclasses if batch is triggered by others
• AbstractTrigger(s) subclasses if batch is triggered at regular periods

In this method, you will be able to other parameters of the batch:

• number of threads
• size of Data blocks retrieved from Database
• Max Job (Number of blocks in memory)

...

Characteristics of a batch

In this part, we will explain the way to declare and configure your batch, mainly by the registerAsBatch method. A batch belongs to a group of batch and is identified by a unique name.

• The (Group name, Batch name) must be unique accross the Batch server (so put in the group name the ID of your project)
• The batch name must not contain any '.'
• Don't catch any exception in registerAsBatch() method, throw an <font size="2" face="Courier New">IllegalStateException.

Add to your batch a method :

public class YourBatchClass extends Traitement 
{
public static final void registerAsBatch() 
{ 
	try { 
		// Old configuration code 
		without any catch block 
		... 
 
	} 
	catch(Exception e) 
	{ 
		Logger logger = Logger.getLogger(getClass()); logger.error( 
				"Registration error for batch "+<BATCH NAME>+", message:"+ e.getMessage(), e); 
		throw new IllegalStateException("Registration error for batch " +  
				"Registration error for batch "+<BATCH NAME>+", message:"+ e.getMessage()); 
 
	} 
}
}

In the registerAsBatch, add the following lines :

TraitementConfiguration lanceur = new TraitementConfiguration();
lanceur.setGroup(<GROUP_NAME>);
lanceur.setName(<BATCH_NAME>);
lanceur.setBatchAppender(new JdbcErrorAppender());
lanceur.setTrtClassName(<Full Class name of the batch>);

Starting request

The starting request is useful to retrieve a resultset of datas, which will be treated by the batch. This will retrieve your basic datas, that will be shared between each thread.

This request is precised in the registerAsBatch method.

StringBuffer strBuf =  new StringBuffer(200);
strBuf.append("select dem_id_demande, dr.tti_num_type_tiers_tir, ");
strBuf.append("dr.tir_num_tiers, dr.tir_sous_num_tiers, dr.qba_id_qba ");
strBuf.append("from demande_requete dr ");
strBuf.append("inner join w_scheduler ws on ( ");
strBuf.append("ws.tti_num_type_tiers_tir = dr.tti_num_type_tiers_tir ");
strBuf.append("and ws.tir_num_tiers_tir = dr.tir_num_tiers ");
strBuf.append("and ws.tir_sous_num_tiers_tir = dr.tir_sous_num_tiers ");
strBuf.append(") ");
strBuf.append("where SCH_NOM_PROC ='");
strBuf.append(IBatchAnalyse.C_TRAITEMENT_EPURER_DIAG_ARTICLE);
strBuf.append("' and (dr.dem_date_exec nvl(dr.dem_delai_effectif_purge,7) < sysdate-1) ");
DaoSqlBatchStart starter = 
    new DaoSqlBatchStart("jdbc/stcom", strBuf.toString(),50,false,
    new String[] {DoDemandeRequete.class.getName()});
lanceur.setBatchStart(starter);

In the last instruction, you define that each thread will work on an array of maximum 50 records (ie a block). Once a block has been treated, the Batch Server will pass the next 50 records to an idle thread.

Configuration of multithreading

Datas from the resultset retrieved by the starting request will be shared between each thread.

By example, the starting request retrieves 220 elements.

In the registerAsBatch, you have to declare the number of threads you want to run :

lanceur.setMaxThreads(3);   // 3 threads in parallel will run and work on datas from the starting request

You also have to declare how to retrieve datas for the starting request. The minJob parameter defines the minimum number of blocks to keep in the waiting pile. If this number is too important, the batch will need more memory as necessary. If this number is too low, there is a risk to have an inactive thread during the time of retrieving of new blocks. The maxJob parameter defines the maximum number of blocks in memory. If this number is too high, there is a risk of OutOfMemory errors. Declare minJob and maxJob :

lanceur.setMaxJob(12);
lanceur.setMinJob(8);

The maximum number of line in memory is maxJob * [nb records / thread] * nb Thread

If the batch has several threads:

• the batch can not contain any non threadsafe instance variable, except the logger.
• the batch must not work on same datas (to avoid database locks).

Triggers

A trigger allow you to precise when a batch have to run. It is an object that you combine with your batch. It is to note that a batch can be activated(started) by several triggers, but mutually, a trigger cannot activate(start) several batches.

The CronTrigger is based on Calendar-like schedules. When you need a Job executed every day at 10:30 a.m., except on Saturdays and Sundays, then a CronTrigger should be used. As the name implies, the CronTrigger is based on the Unix cron expression.

http://www.opensymphony.com/quartz/api/org/quartz/CronTrigger.html

To add a trigger, declare a new trigger as below :

// Trigger
CronTrigger trigger = new CronTrigger();
 
// Fire every day at 4h20
trigger.setCronExpression("0 20 4 ? * SUN");
trigger.setStartTime(new Date());
// Always valid
trigger.setEndTime(null);

Link the trigger with your batch

trigger.setGroup(IBatchPurge.C_GROUPE_TRIGGERS);                             // trigger group
trigger.setName(IBatchPurge.C_TRIGGER_JOURNAL_PURGE_TREATMENT);   // trigger name
trigger.setJobGroup(IBatchPurge.C_GROUPE); // batch group
trigger.setJobName(IBatchPurge.C_TREATMENT_JOURNAL_PURGE_TREATMENT);   //   batch name

and declare the trigger in the scheduler :

lanceur.setTriggers(new Trigger[] { trigger });

You can add several triggers for the same batch in your scheduler.

• The couple (trigger name - trigger group) must be unique by trigger accross all batches.
• The quadruplet batch name - batch group - trigger name - trigger group must be unique by trigger.
• The batch name can not contain any '.'

By default, the timezone of your trigger is the one of the server on which the batch console is installed. (ex : Europe/Paris)

Linkers

Sometimes, it can be interesting to chain some batches. Linkers allow you to accomplish it without much effort.

In the batch console, we can declare two types of link:

• AndLinker : This component allow to define a set of batches having to be executed before launching the current batch.
• OrLinker : This component allow to define a set of batches whose one of them has to be executed before launching the current batch

To declare an AndLinker, you have to code the following lines :

AndLinker linker = new AndLinker();
// declare the batches that must be executed to trigger this batch
linker.addNecessaryBatch(IBatchAppro.C_GROUPE, IBatchAppro.C_TRAITEMENT_SUIVI); 
// to execute the current batch

To declare an OrLinker, the lines to code are the following :

OrLinker linker = new OrLinker();
linker.addNecessaryBatch(IBatchAppro.C_GROUPE, IBatchAppro.C_TRAITEMENT_BESOIN_MIN);
linker.addNecessaryBatch(IBatchAppro.C_GROUPE, IBatchAppro.C_TRAITEMENT_BESOIN_MAX);
linker.addNecessaryBatch(IBatchAppro.C_GROUPE, IBatchAppro.C_TRAITEMENT_IMPL);
lanceur.setLinkers(new OrLinker[] { linker });

In both cases, link the linkers to your scheduler :

lanceur.setLinkers(new AndLinker[] { linker }); for AndLinkers

Registration

Starting from 5.0.0.0, just return the TraitementConfiguration object.

Before 5.0.0.0, call the following method to register your batch :

// Here we don't want the batch to be activated on deployment, 
// in this case, it must be manually activated
// By default, batches are enabled
lanceur.disable();   
lanceur.register();
lanceur.save();

• Remember that when a deployment of bar will occur, your batch will have the state you have defined in the registration code(enabled or disabled) overriding the previous state of the Batch.

Authors

• David Dégardin
• Philippe Mouawad

History

• Page created by Pmouawad, 5 November 2007: New page: {| width=&quot;904&quot; | width=&quot;38&quot; valign=&quot;top&quot; | '''V 1&lt;br /&gt; 06/07''' | width=&quot;850&quot; | = '''Development of a batch''' = == 1 - Type of batch == ==...
• Last modified by Pmouawad, 11 June 2009: /* Registration */