Database management is a pain. Database migration is a major pain – and Amazon Web Services (AWS) knows it. The hyperscaler enabled its users to migrate their data without too much blood, sweat and tears with its 2016's AWS Database Migration Service (AWS DMS), but now the company is aiming to make it even easier and cheaper with AWS DMS Serverless.
The problem it addresses is that to migrate databases, engineers must do a lot of grunt work. Besides cleaning the database of duplicate and incomplete data, they must also work out capacity planning to avoid bottlenecks. Then, when they finally start the database import or data ingestion, they must continuously monitor and manually scale capacity to maintain optimal performance.
Easing up on the guesswork
AWS DMS Serverless attempts to eliminate some of these problems. In particular, it addresses the need for guesswork in determining the required compute resources and manages the operational load. It also automates capacity provisioning, scaling and optimization.
This enables, in theory, a high-performance, uninterrupted migration. Mea culpa, I've overseen DBMS migrations over the years. None of them have been uninterrupted. Still, anything that simplifies and streamlines database migrations, while making the process more cost-effective by automatically establishing, scaling and managing migration resources is a good thing.
This new service supports Microsoft SQL Server, PostgreSQL, MySQL and Oracle as data sources. This serverless option also enables data targets, ranging from Amazon Aurora and Amazon Relational Database Service (Amazon RDS) to Amazon S3, Amazon Redshift and Amazon DynamoDB, among others. Looking ahead, AWS DMS Serverless will add new data sources and target support.
Engineers have their say
This diversity of sources and targets opens numerous opportunities. AWS DMS Serverless enables the migration of databases and helps build modern data strategies. AWS also claims it can help synchronize ongoing data replications into data lakes or data warehouses from a multitude of disparate data sources.
Faiz Khan at Wanclouds, a multi-cloud managed service provider company, worries, however, that, "while it seems pretty intuitive that serverless will reduce costs, it comes with its own challenges — including being locked into AWS. Serverless databases, and migration to them, would be a tiny subset of the overall cloud migration market."
On the other hand, Piyush Tripathi, a tech lead at Square, recalls a previous Twilio Covid-19 contact tracing project where "a really time-consuming part of it was the scaling of operations and shifting to AWS database systems."
Tripathi continued: "Had AWS DMS Serverless been available, it seems like it would have been just the ticket for dealing with these challenges. It would have made the whole process of database migrations a whole lot easier and lighter on the pocket. If we'd had this tool earlier, we could have spent more time on the strategic stuff, leaving the nitty-gritty, techy part of database scaling and migration to AWS DMS Serverless."
Of course, the new AWS program can't do everything. For example, an Amazon engineer told me, "There's only 100GB allocated storage available for data replication. For larger database replications, you must partition your workload into separate serverless replications."
AWS DMS Serverless isn't the only such program. There are numerous other programs from companies like Fivetran, Veeam Backup & Replication and Oracle Goldengate.
Migration headaches
Still, migrating data is one headache. Integrating data from disparate sources is another. The first, extract, transform, load (ETL), is simply importing data. In the second, extract, load, transform (ELT), the data is polymorphed after it's delivered to its destination, such as a cloud data warehouse.
AWS DMS Serverless is an ETL service. It creates a virtual private cloud (VPC) and defines source and target endpoints. It also introduces replication as a concept to replace tasks and instances usually handled by standard AWS DMS.
The service offers three types of replication: Full Load for complete migration of existing data, Change Data Capture (CDC) for replicating data changes and a combination of both for migration and continuous data updates.
AWS DMS Serverless allows users to manage costs by setting maximum capacity. This ensures that the service only consumes what is allocated. AWS DMS Serverless automatically scales capacity down as traffic volume decreases, ensuring customers only pay for what they need. This feature can be adjusted only when the replication process is paused.
So, will AWS DMS relieve the blood, sweat and tears of database migration? The future looks brighter for engineers, it seems – for now, anyway.