I wrote a detailed and technical blog post demonstrating an integration of Spark Structured Streaming with Apache Kafka messages and Snowflake.
An overview of the content is:
- querying Twitter API for realtime tweets
- setting up a Kafka server
- producing messages with Kafka
- consuming and parsing Kafka messages with Spark Structured Streaming
- explanation of the streaming model of Spark Structured Streaming
- upserting latest data to Snowflake
You can find the full blog post here.
A small preview: