Tuesday, 6 November 2012

Talend - Consolidating multiple tflowmeter for insert and delete operation for a job to INSERT_COUNT and DELETE_COUNT for JOB



Problem - You have a job which inserts data into same table via different output components in different flows and also delete data from this table. At the end of load you want to get total count from insert flow and delete flow

Solution - You will add tflowmeter component as required after each output component (remember tflowmeter component gives number of rows flowed to output component and not number of rows processed by database. example number of rows could be 400 but a delete operation might delete only 100 rows from database - so be clear that flowmeter does not gives number of rows actually processed by database).
For each tflowmeter you add same label for kind of operation - like for all insert operation components add label as INSERT and for all delete operation components add label as DELETE. from tflowmetercatcher - insert all rows into a flatfile. So now in your flatfile you will have many rows for each iteration of output component (insert/delete) with each row having label as INSERT/DELETE with count of records processed.
All we need to do now is to use taggregate component to aggregate these rows group by Label so that we get only two rows with DELETE and INSERT as label and SUM(counts)..
and further its up to your requirements to either merge these rows into two separate columns or two rows. See snapshot below







No comments:

Post a Comment