Sunday 27 November 2011

Talend - ETL - Generate job exports from Talend Repository using CommandLine - Export job

Problem - How to generate job exports from Talend Repository using commandline utility from Talend.
Scenario - Lets assume you have your Talend remote repository at linux machine and repository DB on oracle.
Your developers have jobs checked in talend respository named TALEND_REPO.
Now administrator or some automated process needs to export job scripts (which includes executable jar/bat files for jobs) using commandline utility

Solution -
---------
We have used two scripts for this one shell (expJob.sh) and one script which is called by this shell script (expJobdetails.txt).
Exeucte shell script on talend repository server or server which has commandline utility software from talend install on it.

expJob.sh
---------
#!/bin/sh
cd <path of commandLine folder of talend software>
echo "Start of export job"
./TISEE-linux-gtk-x86_64 --disableShellInput -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace scriptFile "<Path of expJobDetails.txt>"

<Path of expJobDetails.txt> - is to be replaced by path of expJobdetails.txt file


expJobDetails.txt
-----------------
initRemote <URL of Talend respository server>
listProject
logonProject -pn <Talend Repository Name> -ul <userLogin> -up <userPassword>
exportJob "<TalendJobName>" -dd <OutputDir> -jc default -af <ExportJobfilename> -jstats
logoffProject
-----------------

How to read expJobDetails.txt
----------------------------------
initRemote - initializes the respository connection
listProject - list all projects inside repository (not required for generating job exports, its just for learning purpose)
logonProject - to logon to repository using userid and password
logoffProject - logoff from project

Replace following in expJobDetails.txt with :
--------------------------------------------------------------------
<URL of Talend respository server> ---> http://localhost:8080/org.talend.administrator
<Talend Repository Name> ---> TALEND_REPO
<userLogin>  ---> Login of user who has access to talend respository (even developers user id who have access to talend repository will work)
<userPassword>  ---> Password of above user
<TalendJobName> ---> Name of Job you want to export
<OutputDir> ---> output directory where exported zip file will be produced
<ExportJobfilename>  ---> Name of zip file to be produced

Wednesday 23 November 2011

Talend - How to capture rejected rows from output component

Problem - You have a job to transfer data from source table to target table and you are using some talend input and output database components for example tMSSqlInput and tMSSqloutput. You are interested in finding the rejected rows which are not processed by your tMSSqloutput component

Solution - For this you can use ROWS-->REJECT property of tMSSqlOutput component. Right click on tMSSqlOutput and select Rows-->Reject and then connect this reject flow to your other tFileOutputDelimited component or similar.
 (Validated on TIS V 4.0.2)