Description

The main business question was to get insights in the performance of 35.000 sensors in the electricity grid, generating TBs of time series data.

The challenge was to:

Extract, Load and Transform (ETL) the sensory data into robust and scalable pipelines.
Schedule the ETL jobs from a centralized server.
Communicate the right descriptive statistics to the stakeholders to facilitate data driven decisions.

The end results are continuously monitored in a hosted dashboard that is refreshed automatically on a daily basis.

Responsibilities

Project lead and stakeholder management.

Python, PySpark SQL, AWS EMR, Apache Airflow, Docker, Gitlab CI/CD, PowerBI.

Forecasting electricity load of congestion points in the electricity grid by combining data from sensors and weather forecasts.

Predicting costly excavation damages and delivering the results automatically to an operational prevention team.

Developing a big data solution including a dashboard for analyzing high volumes of noisy sensory data.

Recently, I have been invited to speak about Data Science developments. I also published scientific papers about my work.