A Hybrid Approach: Why Pure Serverless Isn’t Always the Answer

We’ve made no secret out of the fact that we love serverless. So have lots of other organizations and for good reason.

When you hear us talking about how great serverless is, or read about other companies adopting the technology, you might be wondering if you are behind the times for not adopting 100% serverless architecture.

You want to love serverless, but you could be spending a lot of time and implementation cost creating complex serverless workarounds for implementations that an out-of-the-box solution could handle with low effort. Without the right skills, your technical debt could be piling up while you take multiple iterations to get the solution you need.

At the same time, you can see how great serverless could be so you strive for these complex solutions. How do you bridge the gap between the serverless struggles you’re experiencing in AWS or Azure and technology that offers out-of-the-box solutions/connectors like AWS non-serverless, MuleSoft, Confluent, or Boomi?

Serverless

There are lots of benefits to adopting serverless technology.

• Scalability
• Security
• Reliability
• Flexibility
• And more.

Some IT leaders are pushing to move entirely to serverless. Organizations may be mandating that all greenfield solutions are implemented using the technology. The idea of scaling and growing on-demand generates a lot of excitement.

Unfortunately, sometimes, excitement can lead down the wrong path. There are good reasons why you may not want to become 100% serverless. The serverless architecture dilemma is almost always a build vs. buy decision. There are tools in AWS’s arsenal that would allow for a low startup cost that you could code for with serverless but at the expense of increased complexity and brittle code that lacks robustness.

Building means a higher implementation cost, but a lower ongoing cost.

Buying gets you lower implementation costs, but higher ongoing cost.

It may not even be merely a case of hard or opportunity costs. Your team might not have the skills to implement the complex solution you need on a serverless stack. If you’re trying to enable and leverage diverse teams in your organization, complex serverless solutions probably isn’t the right path.

It could be time to take a hard look at your use cases. Some of them just may not be suitable for serverless.

At Big Compass, we always believe that you should be using the right tool for the job. Forcing serverless isn’t always a good idea. As the saying goes, “just because you can do a thing, doesn’t mean you must do a thing.” The technology’s inherent flexibility allows you to code almost any solution, but adding unnecessary complexity to your solutions isn’t in your company’s best interests.

Hybrid is the Answer

If you’re trying to force serverless in the following use cases, it’s a good indication that you may want to take a step back to reassess:

• Subscribing to an external JMS queue
• Very large transformations
• Web and mobile apps
• Subscribing to events in other systems such as Salesforce Platform Events
• Very large batches of data
• Massive data enrichment
• Orchestration

These are just a few of the use cases where building a serverless solution can be more burdensome than using an existing, non-serverless tool or connector.

That doesn’t mean you need to abandon serverless, however. There are hybrid alternatives to a variety of use cases that veer away from 100% serverless but still let you take advantage of its many benefits.

Below you’ll find hybrid alternatives for a few of these use cases — and these are just the start.

Subscribing to a Queue or Topic Outside of the AWS Ecosystem

Let’s say that your organization mandates the use of Confluent or Kafka for its messaging backbone. To subscribe to a Kafka/Confluent topic in a 100% serverless environment, you’d need to create a Lambda function that would be “always-on” to subscribe to Kafka topic messages.

Because the solution is always on, you’ve added unnecessary complexity to the solution and started to shed some of the benefits you get from serverless such as allowing your Lambda to go to sleep to save cost.

Hybrid alternative: Instead, you could use an AWS EC2 and the Kafka SDK to listen for messages coming through the Kafka topics. We talk about this example more in our recent webinar Serverless Cloud Integrations: The Path Forward.

Very Large Transformations or File Handling

Transformations can also be a challenge. For instance, if you have a 10 GB file that needs to be transformed for downstream processing, that’s a problem. To begin with, Lambdas can only handle up to 10 GB of memory.

You could write a streaming service that chunks the large file and transforms it in an AWS Lambda, but that can become very complex which increases risk and chance for error. Plain ol’ NodeJS or Java can execute transformations, but some tools are purpose-built for large, complex transformations.

Hybrid Alternative: You can use AWS EMR, which was created to handle very large data sets at scale. It efficiently handles a transform like our example and will pump your data into a desired AWS service like S3 for downstream processing.

Web and Mobile Applications

Powering web and mobile application bookends are a great use of serverless. Your front end could be a static site on S3 to make your front end serverless. AWS Fargate is another avenue that would make your application serverless on the front end. But this isn’t always the best or most efficient solution depending on your organization’s skillset.

Hybrid Alternative: Deploy your code using Elastic Beanstalk to a load-balanced, highly available, auto-scaling EC2 cluster. This is a tried and true method without loading additional overhead. While Fargate is a great option, you may not fully understand the nuances of how a serverless front end can work or have the staff and time to learn it.

Subscribing to Events in a Common System

Like subscribing to a JMS queue, it’s tough to subscribe to events that happen in another system, like Salesforce. Traditional serverless services don’t offer the ability, natively, to subscribe to Salesforce Platform Events and PushTopics.‍

Hybrid Alternative: Use AWS AppFlow. AppFlow can subscribe to Salesforce events, out of the box, and send the data to the desired AWS service or another system, like Snowflake. This lets you ingest your data with an inbound endpoint that can be stood up in minutes on AWS. You can then use your serverless services downstream for processing.

Conclusion

Serverless is awesome. That’s why we love it. However, the organization that can effectively get to a 100% serverless footprint is rare. And since serverless is still a relatively new technology, it’s possible in the future that 100% serverless footprints become more common with a growing serverless feature set. For now, sometimes it makes sense to combine the power of serverless with the tools that make your job easier and lifts some of your team’s burden. Hybrid architectures are very common and can be the best path forward.

If you’re wondering how you can marry the benefits of serverless with the right tools to meet your specific use cases, reach out to Big Compass. We’re happy to help you find the balance between the two.