Advertisements
Category Archives: Data Engineering
Tips for easier development on AWS
I wanted to share a couple of tips for easier development on AWS. 1. Use a local version of AWS for development and testing Try something like localstack to stand up a local AWS environment. This will run AWS API compliant … Continue reading
Posted in Big Data, Data Engineering
Leave a comment
Apache Spark Presentation
I’ve published online the presentation on Apache Spark I’ve made for a introductory lecture to graduate students at Maastricht University. If interested, please take a look at the presentation here.
Posted in Big Data, Data Engineering, Data Systems
Leave a comment
My articles for Sonra Intelligence
Apache Airflow Using Apache Airflow to build reusable ETL on AWS Redshift Apache Kafka + Spark Streaming + Redshift Streaming Tweets to Snowflake Data Warehouse with Spark Structured Streaming and Kafka Advanced Spark Structured Streaming – Aggregations, Joins, Checkpointing Snowflake … Continue reading
Posted in Big Data, Data Engineering, Data Systems, Data Warehousing
Leave a comment
Loading Data into Snowflake Data Warehouse and performance of joins
I wrote a detailed article showing how to load 6GB of data into Snowflake using the PUT and COPY INTO commands. Then I evaluated the performance of joins and how caching and instance size affects them. You find the full … Continue reading
Posted in Data Engineering, Data Systems, Data Warehousing
Leave a comment
Caching in Snowflake Data Warehouse
I wrote a technical article covering how Snowflake uses caching on several layers (virtual warehouses caching data and caching of result sets). In the article I also explain how this works and what are the benefits of caching. You can read … Continue reading
Posted in Data Engineering, Data Systems, Data Warehousing
Leave a comment
Using Spark Structured Streaming to upsert Kafka messages into a database
I wrote a detailed and technical blog post demonstrating an integration of Spark Structured Streaming with Apache Kafka messages and Snowflake. An overview of the content is: querying Twitter API for realtime tweets setting up a Kafka server producing messages … Continue reading
Posted in Big Data, Data Engineering, Python
Leave a comment
Advanced Spark Structured Streaming – Aggregations, Joins, Checkpointing
I wrote a blog post demonstrating advanced Spark Structured Streaming topics. An overview of the content is: setting up a Kafka server producing messages with Kafka consuming tweets with Spark Structured Streaming watermarking messages parsing JSON data performing aggregattion queries … Continue reading
Posted in Big Data, Data Engineering, Python
Leave a comment
