Talend Open Studio: How to set up context variables

Talend Open Studio: How to set up context variablesA common concept of making your ETL process easy to configure is to use global variables. This caters for scenarios when you have to move your ETL processes from development to testing and then to production. The idea is to change a few variable values in a central file and then your ETL process will already run on another environment: What a time saver!In this tutorial we will take a look at how...

Read More

Talend Open Studio: Populating a date dimension

Populating a date dimensionDate dimensions are an essential part of a data warehouse. Usually they are only populated once. Scripts can be created on the database side (as outlined here), but if you are working on various projects involving a variety of databases, it is more efficient to create just one ETL job which can be used to populate any database.In this tutorial we will have a look at creating such an ETL job with Talend Open Studio for...

Read More

Talend Open Studio: Scheduling and command line execution

Talend Open Studio: Scheduling and command line executionIn this tutorial we will take a look at how to export a Talend Open Studio ETL job to an autonomous folder and schedule the job via crontab. In order to follow this tutorial, the reader should be familiar with the basic functionality of Talend Open Studio for Data Integration. How to export a jobRight click on your job and choose Export job.In the export settings define:the export...

Read More

Talend: Setting up database logging for a project

Talend: Setting up database logging for a projectWhen executing Talend ETL jobs it is quite important to store the logging information. This is not only relevant for the last execution of an ETL job, but keeping a longer logging history can be quite an advantage. This logging information can be stored in flat files or in a database. We will have a look at the latter option here. I will not go into too much detail, but provide a quick overview....

Read More

PostgreSQL: Auto generating a sample dataset

Sometimes you just want to create simple sample datasets for tables quickly. If you are lucky your database provides some native SQL functions for this purpose.PostgreSQL has quite an interesting recursive WITH statement which allows you to create a loop: You can define in example a value that is incremented with each iteration until a certain condition is met. This comes in quite handy when creating sample data for a date dimension in example:Let's...

Read More