Skip to content

migration

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

migration

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
import_export.rb
import_export.rb
 
 

README.md

Migrating data from RethinkDB 1.7 and above

RethinkDB 1.7+ ships with rethinkdb dump and rethinkdb restore, which provide a smoother migration experience: see the complete guide on rethinkdb.com for information on how to use these commands.

Migrating data from RethinkDB 1.6.x and earlier (deprecated)

Due to improvements in the data format, you will need to migrate your existing data files before upgrading to new versions of RethinkDB.

You can use this handy data migration script to easily migrate your cluster to a new version.

Steps for migrating your cluster

  1. Run import_export.rb --export --host HOST --port PORT, where HOST:PORT connects to a machine in your cluster. This will write everything in your cluster to disk, so make sure you have enough space.
    Note: The port is the port for clients (by default 28015).

  2. Upgrade RethinkDB on all your servers, then move or delete all your rethinkdb_data and data directories (they are incompatible with the new version).

  3. Run import_export.rb --import --host HOST --port PORT. This will re-import all the data on disk into your cluster. You must run this in the same directory you ran the import_export.rb --export command, or else specify the directory to read from with import_export.rb --import DIR. You will also have to specify the directory if you exported from more than one RethinkDB cluster.

NOTE: If any of your tables use a nonstandard primary key (anything except id), you need to tell the script about the nonstandard keys while importing. (We stupidly forgot to build a way to fetch the primary key of a table into the original RDB protocol; this will be fixed in future versions.) You can specify this with the --primary-keys option, which takes a JSON document of the form {"db_name.table_name": "key_name, ...}.

Required dependencies

The migration script requires Ruby to be installed, as well as a few gems:

gem install json ruby_protobuf

Note for OS X users: Due to an incompatibility between ruby_protobuf and Ruby 1.8.7, Ruby 1.9 is required for the migration script.

Simple example

Alice is running a single RethinkDB server on localhost with the default ports. She does the following:

# Make a directory for the migration files we'll generate.
~ $ mkdir rethinkdb_migration
~ $ cd rethinkdb_migration
# Fetch the migration script from Github.
~/rethinkdb_migration $ curl -L \
                          http://raw.github.com/rethinkdb/rethinkdb/next/scripts/migration/import_export.rb \
                          > import_export.rb
# Run the migration script.
~/rethinkdb_migration $ ruby import_export.rb --help
Usage: import_export.rb [options]
    -h, --host HOST                  host to connect to (default localhost)
    -p, --port PORT                  port to connect on (default 28015)
    -e, --export                     export data from RethinkDB into new directory
    -i, --import [DIR]               import data into RethinkDB from directory DIR
    -k, --primary-keys JSON_SPEC     when importing data from old RethinkDB versions, you
                                     must specify all non-`id` primary keys as follows:
                                     '{"db_name.table_name": "key_name"}'
~/rethinkdb_migration $ ruby import_export.rb --export
2013-03-14T20:49:59-07:00 Exporting to rethinkdb_export_2013-03-14T20:49:59-07:00_23642...
2013-03-14T20:49:59-07:00 Connecting to localhost:28015...
2013-03-14T20:49:59-07:00 Connected with query language Query_Language_1...
2013-03-14T20:49:59-07:00 Exporting database test3...
2013-03-14T20:49:59-07:00 Exporting database test...
2013-03-14T20:49:59-07:00 Exporting table test.test2...
2013-03-14T20:49:59-07:00 Exporting table test.foo...
2013-03-14T20:49:59-07:00 Exporting table test.test...
2013-03-14T20:49:59-07:00 Exporting database test2...
2013-03-14T20:49:59-07:00 Exporting table test2.abc...
2013-03-14T20:49:59-07:00 Done!
~/rethinkdb_migration $ <SHUT DOWN RETHINKDB>
~/rethinkdb_migration $ <MOVE/DELETE RETHINKDB_DATA AND DATA DIRECTORY IN INSTANCE DIRECTORY>
~/rethinkdb_migration $ <UPGRADE RETHINKDB>
~/rethinkdb_migration $ <RESTART RETHINKDB>
~/rethinkdb_migration $ ruby import_export.rb --import
2013-03-14T20:50:31-07:00 No directory specified, defaulting to `rethinkdb_export_2013-03-14T20:49:59-07:00_23642`.
2013-03-14T20:50:31-07:00 Importing from rethinkdb_export_2013-03-14T20:49:59-07:00_23642...
2013-03-14T20:50:31-07:00 Connecting to localhost:28015...
2013-03-14T20:50:31-07:00 Connected with query language Query_Language_2...
2013-03-14T20:50:31-07:00 Importing database test3...
2013-03-14T20:50:31-07:00 Importing database test...
2013-03-14T20:50:31-07:00 Importing table test.test2 with primary key id...
2013-03-14T20:50:31-07:00 Importing table test.foo with primary key id...
2013-03-14T20:50:31-07:00 Importing table test.test with primary key id...
2013-03-14T20:50:31-07:00 Importing database test2...
2013-03-14T20:50:31-07:00 Importing table test2.abc with primary key id...
2013-03-14T20:50:32-07:00 Done!
~/rethinkdb_migration $

Complex example

Bob is running two distinct RethinkDB clusters. He can connect to one of them at newton:60715 and the other at magneto:60515. The cluster on newton has two tables test.test and test.foo that use as primary key pkey (instead of id).

# Make a directory for the migration files we'll generate.
~ $ mkdir rethinkdb_migration
~ $ cd rethinkdb_migration
# Fetch the migration script from Github.
~/rethinkdb_migration $ curl -L \
                          http://raw.github.com/rethinkdb/rethinkdb/next/scripts/migration/import_export.rb \
                          > import_export.rb
# Run the migration script.
~/rethinkdb_migration $ ruby import_export.rb --help
Usage: import_export.rb [options]
    -h, --host HOST                  host to connect to (default localhost)
    -p, --port PORT                  port to connect on (default 28015)
    -e, --export                     export data from RethinkDB into new directory
    -i, --import [DIR]               import data into RethinkDB from directory DIR
    -k, --primary-keys JSON_SPEC     when importing data from old RethinkDB versions, you
                                     must specify all non-`id` primary keys as follows:
                                     '{"db_name.table_name": "key_name"}'
~/rethinkdb_migration $ ruby import_export.rb -h newton -p 60715 --export
2013-03-14T21:22:01-07:00 Exporting to rethinkdb_export_2013-03-14T21:22:01-07:00_5048...
2013-03-14T21:22:01-07:00 Connecting to newton:60715...
2013-03-14T21:22:01-07:00 Connected with query language Query_Language_1...
2013-03-14T21:22:01-07:00 Exporting database test2...
2013-03-14T21:22:01-07:00 Exporting table test2.abc...
2013-03-14T21:22:01-07:00 Exporting database test3...
2013-03-14T21:22:01-07:00 Exporting database test...
2013-03-14T21:22:01-07:00 Exporting table test.test2...
2013-03-14T21:22:01-07:00 Exporting table test.test...
2013-03-14T21:22:01-07:00 Exporting table test.foo...
2013-03-14T21:22:01-07:00 Done!
~/rethinkdb_migration $ ruby import_export.rb -h magneto -p 60515 --export
2013-03-14T21:22:47-07:00 Exporting to rethinkdb_export_2013-03-14T21:22:47-07:00_5052...
2013-03-14T21:22:47-07:00 Connecting to magneto:60515...
2013-03-14T21:22:47-07:00 Connected with query language Query_Language_1...
2013-03-14T21:22:47-07:00 Exporting database test2...
2013-03-14T21:22:47-07:00 Exporting table test2.abc...
2013-03-14T21:22:47-07:00 Exporting database test3...
2013-03-14T21:22:47-07:00 Exporting database test...
2013-03-14T21:22:47-07:00 Exporting table test.test2...
2013-03-14T21:22:47-07:00 Exporting table test.test...
2013-03-14T21:22:47-07:00 Exporting table test.foo...
2013-03-14T21:22:47-07:00 Done!
~/rethinkdb_migration $ ls
import_export.rb
rethinkdb_export_2013-03-14T21:22:01-07:00_5048
rethinkdb_export_2013-03-14T21:22:47-07:00_5052
~/rethinkdb_migration $ <SHUTDOWN BOTH CLUSTERS>
~/rethinkdb_migration $ <MOVE/DELETE ALL RETHINKDB_DATA AND DATA DIRECTORY IN INSTANCE DIRECTORIES>
~/rethinkdb_migration $ <UPGRADE BOTH CLUSTERS>
~/rethinkdb_migration $ <RESTART BOTH CLUSTERS>
~/rethinkdb_migration $ ruby import_export.rb -h newton -p 60715 --import rethinkdb_export_2013-03-14T21:22:01-07:00_5048 \
                          --primary-keys '{"test.test": "pkey", "test.foo": "pkey"}'
2013-03-14T21:25:14-07:00 Importing from rethinkdb_export_2013-03-14T21:22:01-07:00_5048...
2013-03-14T21:25:14-07:00 Connecting to newton:60715...
2013-03-14T21:25:14-07:00 Connected with query language Query_Language_2...
2013-03-14T21:25:14-07:00 Importing database test2...
2013-03-14T21:25:14-07:00 Importing table test2.abc with primary key id...
2013-03-14T21:25:14-07:00 Importing database test3...
2013-03-14T21:25:14-07:00 Importing database test...
2013-03-14T21:25:14-07:00 Importing table test.test2 with primary key id...
2013-03-14T21:25:15-07:00 Importing table test.test with primary key pkey...
2013-03-14T21:25:15-07:00 Importing table test.foo with primary key pkey...
2013-03-14T21:25:15-07:00 Done!
~/rethinkdb_migration $ ruby import_export.rb -h magneto -p 60515 --import rethinkdb_export_2013-03-14T21:22:47-07:00_5052
2013-03-14T21:25:48-07:00 Importing from rethinkdb_export_2013-03-14T21:22:47-07:00_5052...
2013-03-14T21:25:48-07:00 Connecting to magneto:60515...
2013-03-14T21:25:48-07:00 Connected with query language Query_Language_2...
2013-03-14T21:25:48-07:00 Importing database test2...
2013-03-14T21:25:48-07:00 Importing table test2.abc with primary key id...
2013-03-14T21:25:48-07:00 Importing database test3...
2013-03-14T21:25:48-07:00 Importing database test...
2013-03-14T21:25:48-07:00 Importing table test.test2 with primary key id...
2013-03-14T21:25:48-07:00 Importing table test.test with primary key id...
2013-03-14T21:25:48-07:00 Importing table test.foo with primary key id...
2013-03-14T21:25:49-07:00 Done!