DynamoDB Table Migration Between Accounts

If you’re like me, you have a love-hate relationship with DynamoDB. It’s an excellent and simple service that can scale (to Petabytes!) and can handle high throughout — the pain that comes with it is that it’s hard to move data around.

Let’s assume you have more than one AWS account, which most big companies have. Now, let’s say you want to move a table — a big table — above 1TB of data. What are our options?

The first option is to use AWS data pipelines(1). I’m not a fan of this solution since it doesn’t let me solve the problems myself. If you ask AWS support, they will surely answer, but I prefer something that I have more control over. Another option is Glue(2), an excellent service, but it also forces you to use Glue crawler, which is great but doesn’t do so well on big tables, and recently we got a new option with API/console(3) that can restore a DDB table from another account (using AWS Backup).

In this blog post, I will show you another way to copy a table using Spark on EMR with a “little” project from AudienceProject, which took AWS’s DynamoDB Spark driver and enhanced it. This gives you much more than simply copying a table, and I will show another usage at the end.

Leave a Comment