Category Archives: Data Engineering

Imports in Python

Python is a great language with a beautiful syntax, lots of useful libraries and great community. Unfortunately, one of the things I like less is the way imports work. In almost every large Python project I do, I want to … Continue reading

Posted in Data Engineering, Python | Leave a comment

Tips for easier development on AWS

I wanted to share a couple of tips for easier development on AWS. 1. Use a local version of AWS for development and testing Try something like localstack┬áto stand up a local AWS environment. This will run AWS API compliant … Continue reading

Posted in Big Data, Data Engineering | Leave a comment

Apache Spark Presentation

I’ve published online the presentation on Apache Spark I’ve made for a introductory lecture to graduate students at Maastricht University. If interested, please take a look at the presentation here.

Posted in Big Data, Data Engineering, Data Systems | Leave a comment

My articles for Sonra Intelligence

Apache Airflow Using Apache Airflow to build reusable ETL on AWS Redshift Apache Kafka + Spark Streaming + Redshift Streaming Tweets to Snowflake Data Warehouse with Spark Structured Streaming and Kafka Advanced Spark Structured Streaming – Aggregations, Joins, Checkpointing Snowflake … Continue reading

Posted in Big Data, Data Engineering, Data Systems, Data Warehousing | Leave a comment

Loading Data into Snowflake Data Warehouse and performance of joins

I wrote a detailed article showing how to load 6GB of data into Snowflake using the PUT and COPY INTO commands. Then I evaluated the performance of joins and how caching and instance size affects them. You find the full … Continue reading

Posted in Data Engineering, Data Systems, Data Warehousing | Leave a comment

Caching in Snowflake Data Warehouse

I wrote a technical article covering how Snowflake uses caching on several layers (virtual warehouses caching data and┬ácaching of result sets). In the article I also explain how this works and what are the benefits of caching. You can read … Continue reading

Posted in Data Engineering, Data Systems, Data Warehousing | Leave a comment

Using Spark Structured Streaming to upsert Kafka messages into a database

I wrote a detailed and technical blog post demonstrating an integration of Spark Structured Streaming with Apache Kafka messages and Snowflake. An overview of the content is: querying Twitter API for realtime tweets setting up a Kafka server producing messages … Continue reading

Posted in Big Data, Data Engineering, Python | Leave a comment