hubio-sync migrate

Run a one-time data migration from source to destination.

Synopsis

hubio-sync migrate --source <name> --destination <name> [flags]

Description

The migrate command performs a one-time full migration of data from a configured source to a destination. Unlike sync, which performs incremental updates, migrate transfers all data in a single operation.

The command supports:

Full file migration with parallel processing
Real-time progress tracking
Dry-run mode for testing
Graceful cancellation via Ctrl+C
Verbose logging for debugging

Flags

Flag	Short	Required	Description
`--source`	`-s`	Yes	Source configuration name
`--destination`	`-d`	Yes	Destination configuration name
`--name`	`-n`	No	Migration name/label for tracking
`--dry-run`		No	Validate configuration without migrating
`--verbose`	`-v`	No	Enable verbose output
`--workers`	`-w`	No	Number of parallel workers (default: 16)

Examples

Basic Migration

hubio-sync migrate --source local-files --destination s3-archive

Output:

Starting migration...
Source: local-files
Destination: s3-archive

Connecting to source...
Connecting to destination...
Starting file transfer...

Progress: 45.2% (1.2 GB / 2.7 GB)
Files: 1,234 / 2,500
Speed: 125.4 MB/s
ETA: 12s

Migration completed successfully!
Duration: 21s
Files transferred: 2,500
Bytes transferred: 2.7 GB

Named Migration

hubio-sync migrate \
  --source mysql-prod \
  --destination snowflake-analytics \
  --name "Q4-2025-migration"

Dry Run

Test your configuration without actually transferring data:

hubio-sync migrate \
  --source local-files \
  --destination s3-backup \
  --dry-run

Output:

DRY RUN MODE - No data will be transferred

Validating migration...
Source: local-files
Destination: s3-backup

Connecting to source... OK
Connecting to destination... OK

Source contains:
  Files: 2,500
  Total size: 2.7 GB

Configuration valid. Ready to migrate.

Verbose Output

Enable detailed logging for debugging:

hubio-sync migrate \
  --source local-files \
  --destination s3-archive \
  --verbose

Custom Worker Count

Adjust parallel processing for your environment:

# More workers for fast networks
hubio-sync migrate \
  --source local-files \
  --destination s3-archive \
  --workers 32

# Fewer workers to reduce resource usage
hubio-sync migrate \
  --source local-files \
  --destination s3-archive \
  --workers 4

With Custom Config File

hubio-sync migrate \
  --source mysql-prod \
  --destination s3-backup \
  --config /path/to/config.json

Progress Display

During migration, the command displays real-time progress:

Progress: 67.3% (1.8 GB / 2.7 GB)
Files: 1,682 / 2,500
Speed: 142.1 MB/s
ETA: 6s

Field	Description
Progress	Percentage complete with bytes transferred/total
Files	Number of files transferred/total
Speed	Current transfer rate in bytes per second
ETA	Estimated time to completion

Graceful Cancellation

Press Ctrl+C to cancel a running migration. The command will:

Stop accepting new file transfers
Wait for in-progress transfers to complete
Report final statistics
Exit with a non-zero status code

^C
Received interrupt signal. Cancelling migration...
Waiting for in-progress transfers to complete...

Migration cancelled.
Files transferred before cancellation: 1,234
Bytes transferred: 1.2 GB

Exit Codes

Code	Description
0	Migration completed successfully
1	Migration failed with error
130	Migration cancelled by user (Ctrl+C)

Error Handling

If a migration fails, the command reports the error and exits:

Migration failed!
Error: failed to write file data/large-file.zip: connection timeout

Files transferred before failure: 1,234
Bytes transferred: 1.2 GB

Configuration Requirements

Before running a migration, ensure your configuration file contains valid source and destination definitions:

{
  "sources": {
    "local-files": {
      "type": "filesystem",
      "path": "/data/exports"
    }
  },
  "destinations": {
    "s3-archive": {
      "type": "s3",
      "bucket": "company-data-archive",
      "region": "us-east-1",
      "access_key": "AKIAIOSFODNN7EXAMPLE",
      "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    }
  }
}

Use hubio-sync config validate to verify your configuration before running a migration.

Performance Tuning

Worker Count

The --workers flag controls how many files are transferred in parallel:

Default (16): Good balance for most environments
Higher (32-64): Better for fast networks with many small files
Lower (4-8): Reduces memory usage and network congestion

Large File Handling

Files larger than 10 MB are handled specially with streaming progress updates, so you can monitor transfer progress for individual large files.