Advanced Spark Structured Streaming – Aggregations, Joins, Checkpointing

I wrote a blog post demonstrating advanced Spark Structured Streaming topics.

An overview of the content is:

  • setting up a Kafka server
  • producing messages with Kafka
  • consuming tweets with Spark Structured Streaming
  • watermarking messages
  • parsing JSON data
  • performing aggregattion queries on the stream of data
  • analyzing execution plans of queries
  • upserting data to Snowflake
  • checkpointing a structured stream

You can find the full blog post here.

A small preview:

Screen Shot 2018-02-11 at 16.50.58.png

About dorianbg

A Data Engineer based in London, United Kingdom
This entry was posted in Big Data, Data Engineering, Python. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s