Serverless computing, a concept that has started gaining traction in the last two years, is all about shifting focus to applications that require no infrastructure management and consume resources only for the seconds for which they are active. In the public cloud space, serverless usually translates to solutions where the provider dynamically manages the allocation of server resources based on workload requirements. AWS Lambda led the way, with Microsoft Azure Functions (and others) quickly catching up. Pricing for serverless frameworks is generally based on the actual amount of resources consumed by the application, instead of prepurchased capacity. As these serverless solutions for stateless applications gain popularity and adoption in next-generation software architectures, where does that leave the relational database? Still a crucial component for many, if not most, applications.
For the past several years, you have had several robust and well-proven models to choose from when it comes to deploying relational databases: from monoliths through microservices and to platform-as-a-service solutions. You can deploy a single “large” server, running a monolithic or consolidated database powering dozens of applications. You can also choose to rely on a microservices-oriented architecture with a suite of independent, small and modular services, each enabling a unique process and serves a specific business goal. The adoption of cloud solutions also provides you with the ability to deploy your databases via infrastructure-as-a-code and even leverage platform-as-a-service solutions, substantially minimizing the operational overhead and complexity for our databases.
However, all these models still rely on the provisioning of database servers. Be it on-premises, in the cloud, or using PaaS. You provision our database capacity based on predicted workload characteristics that determine the size and configuration of your servers. Sure, you can scale up, scale down, or scale out your database in response to workloads (depending on the database technology used) but this process isn’t meant to be done frequently.
Instead, you are supposed to scale in response to periodic events, such as an upcoming holiday season that will generate additional transactions for your e-commerce application or scale in response to onboarding a big new client for your company’s SaaS product. Having dedicated database servers makes the most sense when workloads are more or less predictable and relatively constant. There could be peaks and dips, but they usually adhere to a predictable pattern. You may need scale your database a couple of times throughout the year, but not multiple times throughout the day. Infrequent database scaling is model best suited for traditional applications.
Next-gen applications introduce next-gen challenges. Some of these workloads can be sporadic, intermittent, and highly unpredictable. For example, bursts of database queries or transactions that might span just minutes or hours each day (or even, each month). Using the same e-commerce app example from earlier, how can your databases provide support for flash sales events without over-provisioning your database servers in advance as a precaution, just in case? Similar challenges exist for other workloads, ranging from online gaming, stock trading, and even analytics (what if you only have a couple of hours a day where your analytics suite generates heavy database load)? Most database administrators will state that you should size your database according to your forecasted peak workload. That’s the conventional wisdom and the correct paradigm when the process of scaling your database is a chore, if even possible.
To leverage the serverless computing paradigm for the database space, you first need to decouple the storage and processing layers of your data architecture. Decoupling storage and compute isn’t exactly a new concept. The idea has already been implemented in the NoSQL and big data analytics space (Amazon EMR, Microsoft’s Azure DLS & DLA, etc.) as well as by various relational database technologies (Oracle RAC, NuoDB), to some extent.
Yet purely decoupling storage and compute isn’t exactly what you’d call serverless. To be fully serverless, your compute should not exist during periods that it does not process data while also providing on-demand autoscaling.
Essentially, deploy a data architecture where the database layer will automatically start, shut down, and scale up/down based on the application workloads, while also abstracting the concept of servers, instances, or clusters. You only need to define a database endpoint and connect your applications; the underlying database technology will scale the storage and compute resources depending on the application needs.
In addition to the benefits in performance and flexibility, serverless database models can also provide high levels of cost-effectiveness. For example, paying on a per-second basis for the database capacity you use and only when the database is active, instead of choosing your database instance size up front.
There are plenty of scalable relational database technologies that offer read or read/write scaling (Oracle RAC, Amazon Aurora, Percona XtraDB, ClustrixDB, NuoDB, etc.). However, these are not natively serverless offerings. There are also innovative solutions for serverless databases, with technologies such as FaunaDB (serverless and globally replicated NoSQL database), Google Cloud Spanner (globally distributed and strongly consistent relational database), or Microsoft Cosmos DB (schema-agnostic multimodel database with a flexible consistency model). But traditional applications wanting to use these database technologies will have to be heavily rewritten or undergo extensive re-platforming. For example, while Google Spanner is a relational database with full ACID functionality (and a fantastic database technology in its own right), it relies on custom client libraries for connectivity and provides a variant of SQL, where transactions are handled by a custom API .
Enabling genuinely serverless and genuinely relational databases, with complete server abstraction/scale on one hand yet maintaining full ANSI SQL and ACID support on the other is a relatively new initiative that’s ripe for innovation; notably, from vendors in the public cloud space (Amazon, Microsoft, Google, and Oracle).
For example, one of the most exciting announcements Amazon made at last year’s Re:Invent conference was a serverless version for the Aurora MySQL database, to be released later in 2018. According to Amazon, Aurora Serverless is designed for “workloads that are highly variable and subject to rapid change, this new configuration allows you to pay for the database resources you use, on a second-by-second basis.” Amazon states that Aurora Serverless users will only pay for the processing when the database is active (as well as for used storage). Amazon is essentially building the database equivalent of an event-driven compute platform. Users provision an endpoint, which acts as a proxy that routes queries to a rapidly scaled fleet of database resources. Based on the information Amazon provides, Aurora Serverless will allow your connections to remain active even as scaling operations take place. Scaling is also supposed to be rapid, with new resources becoming online within seconds.
It’s incredible to see how changes in technology drive changes in development and application deployment patterns. Supporting API-driven operations and scale is becoming progressively more critical for next-generation data architectures, with serverless databases becoming an inherently important component.
One of the holy grails for current RDBMS technologies is to combine the benefits of serverless computing with the flexible relational data model that developers know and the full ANSI SQL and ACID support that existing applications are compatible with. In seems that whichever database vendor reaches the top first and release a fully relational and fully serverless database engine is sure to leave a significant mark on the market. So, Amazon, Oracle, Microsoft, Google, and others, the race to peak database innovation is on!
This article is published as part of the IDG Contributor Network.Want to Join?