This project provides a set of utilities for migrating from Apache Ignite 2 to Apache Ignite 3.

The project is split into two major artifacts:

• migration-tools-cli: Packs submodules into a simple CLI tool.
  • configuration-converter: Converts an Apache Ignite 2 configuration file into its Apache Ignite 3 counterpart.
  • sql-ddl-generator: Generates an SQL DDL script for each table in the provided Apache Ignite 2 configuration.
  • persistence-tools: Allows reading a Apache Ignite 2 node's working directory and migrate the existing data.

The tools will be distributed through maven. Currently, they are in the CI/CD build.

The CLI tools are distributed in a zip and tar.gz packages.

The executable (migration-tools) is in the bin folder.

• Java (any version supported by Ignite 2)
• Posix shell

The migration tools adapter can be imported to your maven project as follows:

<dependency>
  <groupId>org.apache.ignite</groupId>
  <artifactId>migration-tools-adapter</artifactId>
  <version>1.0-SNAPSHOT</version>
</dependency>

Display help for the available commands: migration-tools --help

Convert an Apache Ignite 2 configuration file into Apache Ignite 3 cluster and node configurations: migration-tools configuration-converter resources/ignite-config.0.xml resources/node.conf resources/cluster.conf --include-defaults

Generate an SQL DDL script for each table based on the Apache Ignite 2 file: migration-tools sql-ddl-generator resources/ignite-config.0.xml

Collection of tools to interact with the Apache Ignite 2 node's work directory.

Examples:

Show persistent-data command help: migration-tools persistent-data --help

Lists the caches available in the Apache Ignite 2 node's work directory.

List caches on node1:

migration-tools \
    persistent-data \
    node1/workDir \
    d270287b-b031-4fdb-abe1-6b124b0be9ef \
    resources/configs/ignite-config.xml \
    list-caches

id      name
-1672482954      Country
2100619      City

Show list-caches command help:

migration-tools \
    persistent-data \
    node1/workDir \
    d270287b-b031-4fdb-abe1-6b124b0be9ef \
    resources/configs/ignite-config.xml \
    list-caches \
    --help

Migrates data from an Apache Ignite 2 node's directory into a live Apache Ignite 3 cluster. The migration is performed partition by partition. In case of an error, you may use the saved progress file along with the --resume-from option to skip the partitions successfully migrated in the previous run.

The mode option defines the error handling policy in case of an irreconcilable mismatch between an Apache Ignite 2 record and the target Apache Ignite 3 table:

• ABORT (default): panics and aborts the migration.
• IGNORE_COLUMN: any additional columns/fields in the cache record will be ignored (not migrated to the table; lost). The others will be migrated as usual.
• SKIP_RECORD: The entire cache record will be ignored (not migrated to the table; lost).
• PACK_EXTRA: Any additional columns/fields in the cache record will be serialized to JSON and stored in the __EXTRA__ column. This is an additional column that the tool adds to the table, it is not a native feature.

The --rate-limiter N option limits the number of records migrated per second by using a very basic rate limiter implementation, which may be prone to bursts.

By default, this command will save the progress in the current work directory. --no-save-progress will not record the progress file.

The --resume-from progress-file.json allows resuming the migration from the provided progress file. All the partitions marked as successfully migrated in this file will be skipped.

Examples:

Migrate the cache Country cache, halting on any error using the ABORT mode:

migration-tools \
    persistent-data \
    node1/workDir \
    d270287b-b031-4fdb-abe1-6b124b0be9ef \
    configs/ignite-config.xml \
    migrate-cache \
    Country \
    127.0.0.1:10800 \
    --mode ABORT

The project can be built using maven: ./gradlew build.

Make sure you meet the following dependencies.

This image is necessary for running tests. If the ignite3.docker.version property is set to latest, you must build the Docker image locally:

1. Clone the Apache Ignite 3 repository.

2. Recommended: clone the same revision as declared in the dependencies ignite3.version.commit. Automatically get the revision hash by running this in migration-tools-root:

   ./gradlew :properties | awk -F ': ' '/ignite3.version.commit/ {print $2}'

3. Build the Docker image (for tests only) : ./gradlew clean :packaging:docker -x check.

The end-to-end tests can take significant time to run. You have the option to limit the number of caches for tests using the -Pe2e.testLimiter=10 property, where 10 is the number of cache examples to test.

Some of the tests use an Apache Ignite 2 persistent directory. Although the framework can recreate the cluster, it can take more than 2 hours. As an alternative result, we provide a prebuilt cluster that you can download:

bash .ci/download-n-extract-sample-cluster.sh

Generates a configuration file that informs the interface adapter of the correct classes for mapping tables/caches. This tool inspects the running Apache Ignite 2 cluster to get this information.

Currently, there are no use cases for this module.

• The messages about properties not converted successfully are hard to read.
• Enable the license requirements in the checkStyle plugin: TODO: Enable License
• Remove the checkstyle suppression from the external file.