My target is to provide sample ETL data flow in PL/SQL, to do that, I need to generate sample data. Basically, I will operate on two data sources: 1) orders data and 2) contact form data:

I received business task to prepare orders/sales report:

• Business requirement is, to be able, to show how much we sale from WWW source, in terms of new clients
• Technical requirements are:
  1. To connect: contact_form table, with orders table, by a algorithm:
     • join by email address
     • deduplica...

Airflow might be used for data transformation in python, as well as scheduler to execute SQL statement.

My target is, to set up airflow, to create table with historical employee list, trough PLPGSQL script, refreshed every 15 min.

Imagine we have table employee with 290 rows =>

1) First I'm...

My target is to do basic data transformation using python in Airflow dag.

Below script is divided into 3 tasks:

• 1) connect to postgres database and extract data to dataframe
• 2) transform dataframe
• 3) write log information into csv file, and dataframe results into json file

Data (datafr...