Airflow might be used for data transformation in python, as well as scheduler to execute SQL statement.

My target is, to set up airflow, to create table with historical employee list, trough PLPGSQL script, refreshed every 15 min.

Imagine we have table employee with 290 rows =>

1) First I'm...

My target is to do basic data transformation using python in Airflow dag.

Below script is divided into 3 tasks:

• 1) connect to postgres database and extract data to dataframe
• 2) transform dataframe
• 3) write log information into csv file, and dataframe results into json file

Data (datafr...