Category Archives: Big Data

Implementing the Speed Layer of Lambda Architecture using Spark Structured Streaming

This post is a part of a series on Lambda Architecture consisting of: Introduction to Lambda Architecture Implementing Data Ingestion using Apache Kafka, Tweepy Implementing Batch Layer using Kafka, S3, Redshift Implementing Speed Layer using Spark Structured Streaming Implementing Serving … Continue reading

Posted in Big Data, Data Engineering | 4 Comments

Implementing the Serving Layer of Lambda Architecture using Redshift

This post is a part of a series on Lambda Architecture consisting of: Introduction to Lambda Architecture Implementing Data Ingestion using Apache Kafka, Tweepy Implementing Batch Layer using Kafka, S3, Redshift Implementing Speed Layer using Spark Structured Streaming Implementing Serving … Continue reading

Posted in Big Data, Data Engineering | 4 Comments

Implementing the Batch Layer of Lambda Architecture using S3, Redshift and Apache Kafka

This post is a part of a series on Lambda Architecture consisting of: Introduction to Lambda Architecture Implementing Data Ingestion using Apache Kafka, Tweepy Implementing Batch Layer using Kafka, S3, Redshift Implementing Speed Layer using Spark Structured Streaming Implementing Serving … Continue reading

Posted in Big Data, Data Engineering | 4 Comments

Ingesting realtime tweets using Apache Kafka, Tweepy and Python

This post is a part of a series on Lambda Architecture consisting of: Introduction to Lambda Architecture Implementing Data Ingestion using Apache Kafka, Tweepy Implementing Batch Layer using Kafka, S3, Redshift Implementing Speed Layer using Spark Structured Streaming Implementing Serving … Continue reading

Posted in Big Data, Data Engineering | 4 Comments

Introduction to Lambda Architecture

This post is a part of a series on Lambda Architecture consisting of: Introduction to Lambda Architecture Implementing Data Ingestion using Apache Kafka, Tweepy Implementing Batch Layer using Kafka, S3, Redshift Implementing Speed Layer using Spark Structured Streaming Implementing Serving … Continue reading

Posted in Big Data, Data Engineering | 4 Comments