Friday, 22 February 2013

Talend: Fetch records deleted by tjdbcoutput component (or other database output) in talend and storing it to global variable


Problem -
Fetching number of records deleted/inserted by tjdbcoutput component (or any other tDatabaseoutput comonent) and performing mathematical operation of addition and saving this value to global variable

Solution -
tJDBCOutput_1 - is our output component.
tJDBCOutput_2 - is our another output component.
DEL_ROW_COUNT - is our global variable to store deleted row by above two components

Talend provides NB_Line_Inserted/Deleted/Updated variables for different database output components which we can use after the component has finished.

What we will do is on subJobOk of these components we use java code in tJava component, where we use java functions to get value of these NB_Line* variables and do mathematical operation and store final value to global variable.





Java code is below
globalMap.put("DEL_ROW_COUNT",
Integer.toString((Integer)globalMap.get("tJDBCOutput_1_NB_LINE_DELETED") +
(Integer)globalMap.get("tJDBCOutput_2_NB_LINE_DELETED")));



Talend: Using database sequence (postgres sql) to populate primary key or surrogate in target table using talend





Using database sequence (postgres sql) to populate primary key in target table using talend

Problem – we want to use database sequence to generate unique values for primary key column in the target table during insert.

Solution – Talend provides additional columns option in tPostgresqloutput component, which I believe would be similar for all other tdatabaseoutput components.

picture below shows target table dim_page where i have a primary key as page_skey and i want to populate it using dim_page_seq sequence created at database level.
what  we need to do is use additional columns options with sql expression as the function to get next value of sequence from database and position replace.
Also in input to this tPostgresqloutput component i pass page_skey as some dummy value using tmap, which gets overwritten by replace option.

using database sequence




input schema for above output component is as below (this schema is passed as output of tmap)