Achieving Data Agility With The Combined Strengths of AWS and Confluent

Strengths of AWS

Combining AWS services alone can be a solid solution for your business on its own. When appropriately combined with Confluent, though, those strengths can significantly extend and enhance the functionality of AWS.

  • Amazon EMR can be used if you want to dedicate a cluster to large data transformations
  • Amazon Batch can be used for large batch processes to scale up and down automatically
  • AWS Lambda can be used to execute custom data transformation in parallel at unmatched speed
  • Amazon S3
  • Amazon RDS
  • Amazon ElasticSearch
  • Confluent Log Storage: Confluent logs can be shipped to AWS and stored in ElasticSearch. You can then define KPIs for your systems and visualize performance against those KPIs using ElasticSearch dashboard.
  • Raw message storage and processing: Amazon API Gateway can accept raw messages and store those in S3 so that the data is never lost. Then, as described in the previous section, API Gateway can send the events to Confluent through AWS Lambda.
  • Pairing with SQL: Confluent has a MySQL connector, as well as several others. Events can be taken in on Confluent Topics and then piped to Amazon RDS running MySQL.
  • AWS CodePipeline
  • AWS CodeBuild
  • AWS CodeDeploy
  • AWS CodeCommit
  • AWS Elastic Beanstalk
  • AWS SAM
  • Customer scripts on AWS
  • Amazon EC2
  • Amazon ECS
  • Amazon EKS
  • Amazon EFS
  • Amazon VPC

Strengths of Confluent

You may come at this combination from the other direction; perhaps it’s the value of Confluent that has you considering ways to make it more powerful and valuable for your business. Here are some of the strengths of Confluent and how AWS can enhance them.

  • KTable
  • ksqlDB
  • KTable
  • ksqlDB
  • KStream
  • Topics

Conclusion

Confluent is the epitome of data movement and data streaming capabilities. AWS is the perfect partner for Confluent technology because it rounds out the data agility expertise of Confluent with highly scalable services to support the high demands of real-time streaming. AWS also solves the needs of Confluent in the areas of data storage and data at rest, which are necessary for robust integration strategies.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aaron Lieberman

Aaron Lieberman

Aaron’s passion for technology drives him to find innovative ways to help advance organizations through technology.