Achieving Data Agility With The Combined Strengths of AWS and Confluent

Strengths of AWS

  • Amazon EMR can be used if you want to dedicate a cluster to large data transformations
  • Amazon Batch can be used for large batch processes to scale up and down automatically
  • AWS Lambda can be used to execute custom data transformation in parallel at unmatched speed
  • Amazon S3
  • Amazon RDS
  • Amazon ElasticSearch
  • Confluent Log Storage: Confluent logs can be shipped to AWS and stored in ElasticSearch. You can then define KPIs for your systems and visualize performance against those KPIs using ElasticSearch dashboard.
  • Raw message storage and processing: Amazon API Gateway can accept raw messages and store those in S3 so that the data is never lost. Then, as described in the previous section, API Gateway can send the events to Confluent through AWS Lambda.
  • Pairing with SQL: Confluent has a MySQL connector, as well as several others. Events can be taken in on Confluent Topics and then piped to Amazon RDS running MySQL.
  • AWS CodePipeline
  • AWS CodeBuild
  • AWS CodeDeploy
  • AWS CodeCommit
  • AWS Elastic Beanstalk
  • AWS SAM
  • Customer scripts on AWS
  • Amazon EC2
  • Amazon ECS
  • Amazon EKS
  • Amazon EFS
  • Amazon VPC

Strengths of Confluent

  • KTable
  • ksqlDB
  • KTable
  • ksqlDB
  • KStream
  • Topics

Conclusion

--

--

--

Aaron’s passion for technology drives him to find innovative ways to help advance organizations through technology.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to serve your website on port 80 or 443 using AWS Load Balancers

Blog Post Automation Test

Alt text

Solving complex problems (part 3)

A Collection of Handy Git Commands

Undo the last session saved with Tmux Resurrect

Not All Containers are Created Equal

7 Levels of Using the Zip Function in Python

The Zip Function in Python

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aaron Lieberman

Aaron Lieberman

Aaron’s passion for technology drives him to find innovative ways to help advance organizations through technology.

More from Medium

Will software developers be the new heroes of the analytics world?

HyperLogLog and Beyond (On distinct counting from a data sketching researcher)

Using Amazon DynamoDB to store data at scale

Comsysto’s Fast & Curious

Title image of Comsysto’s Fast & Curious edition