Skip to content

Commit

Permalink
Use go style short options everywhere
Browse files Browse the repository at this point in the history
Also change references to `gh-osc` to `gh-ost`. In README.md there
is one reference left, but that's to describe the history.

Related Issue: github#127
  • Loading branch information
dveeden committed Aug 3, 2016
1 parent 1c1869d commit bf9ecf3
Show file tree
Hide file tree
Showing 7 changed files with 129 additions and 101 deletions.
128 changes: 64 additions & 64 deletions doc/cheatsheet.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,93 +21,93 @@ However even with RBR we suggest this is the least master-intrusive operation mo

```shell
gh-ost \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--throttle-control-replicas="myreplica.1.com,myreplica.2.com" \
--max-lag-millis=1500 \
--user="gh-ost" \
--password="123456" \
--host=replica.with.rbr.com \
--database="my_schema" \
--table="my_table" \
--verbose \
--alter="engine=innodb" \
--switch-to-rbr \
--allow-master-master \
--cut-over=default \
--exact-rowcount \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
[--execute]
-max-load=Threads_running=25 \
-critical-load=Threads_running=1000 \
-chunk-size=1000 \
-throttle-control-replicas="myreplica.1.com,myreplica.2.com" \
-max-lag-millis=1500 \
-user="gh-ost" \
-password="123456" \
-host=replica.with.rbr.com \
-database="my_schema" \
-table="my_table" \
-verbose \
-alter="engine=innodb" \
-switch-to-rbr \
-allow-master-master \
-cut-over=default \
-exact-rowcount \
-default-retries=120 \
-panic-flag-file=/tmp/ghost.panic.flag \
-postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
[-execute]
```

With `--execute`, migration actually copies data and flips tables. Without it this is a `noop` run.
With `-execute`, migration actually copies data and flips tables. Without it this is a `noop` run.


### b. Connect to master

If you don't have replicas, or do not wish to use them, you are still able to operate directly on the master. `gh-ost` will do all operations directly on the master. You may still ask it to be considerate of replication lag.

- Your master must produce binary logs in RBR format.
- You must approve this mode via `--allow-on-master`.
- You must approve this mode via `-allow-on-master`.

```shell
gh-ost \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--throttle-control-replicas="myreplica.1.com,myreplica.2.com" \
--max-lag-millis=1500 \
--user="gh-ost" \
--password="123456" \
--host=master.with.rbr.com \
--allow-on-master \
--database="my_schema" \
--table="my_table" \
--verbose \
--alter="engine=innodb" \
--switch-to-rbr \
--allow-master-master \
--cut-over=default \
--exact-rowcount \
--default-retries=120 \
--panic-flag-file=/tmp/ghost.panic.flag \
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
[--execute]
-max-load=Threads_running=25 \
-critical-load=Threads_running=1000 \
-chunk-size=1000 \
-throttle-control-replicas="myreplica.1.com,myreplica.2.com" \
-max-lag-millis=1500 \
-user="gh-ost" \
-password="123456" \
-host=master.with.rbr.com \
-allow-on-master \
-database="my_schema" \
-table="my_table" \
-verbose \
-alter="engine=innodb" \
-switch-to-rbr \
-allow-master-master \
-cut-over=default \
-exact-rowcount \
-default-retries=120 \
-panic-flag-file=/tmp/ghost.panic.flag \
-postpone-cut-over-flag-file=/tmp/ghost.postpone.flag \
[-execute]
```

### c. Migrate/test on replica

This will perform a migration on the replica. `gh-ost` will briefly connect to the master but will thereafter perform all operations on the replica without modifying anything on the master.
Throughout the operation, `gh-ost` will throttle such that the replica is up to date.

- `--migrate-on-replica` indicates to `gh-ost` that it must migrate the table directly on the replica. It will perform the cut-over phase even while replication is running.
- `--test-on-replica` indicates the migration is for purpose of testing only. Before cut-over takes place, replication is stopped. Tables are swapped and then swapped back: your original table returns to its original place.
- `-migrate-on-replica` indicates to `gh-ost` that it must migrate the table directly on the replica. It will perform the cut-over phase even while replication is running.
- `-test-on-replica` indicates the migration is for purpose of testing only. Before cut-over takes place, replication is stopped. Tables are swapped and then swapped back: your original table returns to its original place.
Both tables are left with replication stopped. You may examine the two and compare data.

Test on replica cheatsheet:
```shell
gh-ost \
--user="gh-ost" \
--password="123456" \
--host=replica.with.rbr.com \
--test-on-replica \
--database="my_schema" \
--table="my_table" \
--verbose \
--alter="engine=innodb" \
--initially-drop-ghost-table \
--initially-drop-old-table \
--max-load=Threads_running=30 \
--switch-to-rbr \
--chunk-size=2500 \
--cut-over=default \
--exact-rowcount \
--serve-socket-file=/tmp/gh-ost.test.sock \
--panic-flag-file=/tmp/gh-ost.panic.flag \
--execute
-user="gh-ost" \
-password="123456" \
-host=replica.with.rbr.com \
-test-on-replica \
-database="my_schema" \
-table="my_table" \
-verbose \
-alter="engine=innodb" \
-initially-drop-ghost-table \
-initially-drop-old-table \
-max-load=Threads_running=30 \
-switch-to-rbr \
-chunk-size=2500 \
-cut-over=default \
-exact-rowcount \
-serve-socket-file=/tmp/gh-ost.test.sock \
-panic-flag-file=/tmp/gh-ost.panic.flag \
-execute
```

### cnf file
Expand All @@ -120,4 +120,4 @@ user=gh-ost
password=123456
```

You may then remove `--user=gh-ost --password=123456` and specify `--conf=/path/to/config/file.cnf`
You may then remove `-user=gh-ost -password=123456` and specify `-conf=/path/to/config/file.cnf`
36 changes: 32 additions & 4 deletions doc/testing-on-replica.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ TL;DR `gh-ost` will make all changes on a replica and leave both original and gh

## Issuing a test drive

Apply `--test-on-replica --host=<a.replica>`.
Apply `-test-on-replica -host=<a.replica>`.
- `gh-ost` would connect to the indicated server
- Will verify this is indeed a replica and not a master
- Will perform _everything_ on this replica. Other then checking who the master is, it will otherwise not touch it.
Expand Down Expand Up @@ -49,17 +49,45 @@ It's your job to:

Simple:
```shell
$ gh-osc --host=myhost.com --conf=/etc/gh-ost.cnf --database=test --table=sample_table --alter="engine=innodb" --chunk-size=2000 --max-load=Threads_connected=20 --initially-drop-ghost-table --initially-drop-old-table --test-on-replica --verbose --execute
$ gh-ost \
-host=myhost.com \
-conf=/etc/gh-ost.cnf \
-database=test \
-table=sample_table \
-alter="engine=innodb" \
-chunk-size=2000 \
-max-load=Threads_connected=20 \
-initially-drop-ghost-table \
-initially-drop-old-table \
-test-on-replica \
-verbose \
-execute
```

Elaborate:
```shell
$ gh-osc --host=myhost.com --conf=/etc/gh-ost.cnf --database=test --table=sample_table --alter="engine=innodb" --chunk-size=2000 --max-load=Threads_connected=20 --switch-to-rbr --initially-drop-ghost-table --initially-drop-old-table --test-on-replica --postpone-cut-over-flag-file=/tmp/ghost-postpone.flag --exact-rowcount --allow-nullable-unique-key --verbose --execute
$ gh-ost \
-host=myhost.com \
-conf=/etc/gh-ost.cnf \
-database=test \
-table=sample_table \
-alter="engine=innodb" \
-chunk-size=2000 \
-max-load=Threads_connected=20 \
-switch-to-rbr \
-initially-drop-ghost-table \
-initially-drop-old-table \
-test-on-replica \
-postpone-cut-over-flag-file=/tmp/ghost-postpone.flag \
-exact-rowcount \
-allow-nullable-unique-key \
-verbose \
-execute
```
- Count exact number of rows (makes ETA estimation very good). This goes at the expense of paying the time for issuing a `SELECT COUNT(*)` on your table. We use this lovingly.
- Automatically switch to `RBR` if replica is configured as `SBR`. See also: [migrating with SBR](migrating-with-sbr.md)
- allow iterating on a `UNIQUE KEY` that has `NULL`able columns (at your own risk)

### Further notes

Do not confuse `--test-on-replica` with `--migrate-on-replica`; the latter performs the migration and _keeps it that way_ (does not revert the table swap nor stops replication)
Do not confuse `-test-on-replica` with `-migrate-on-replica`; the latter performs the migration and _keeps it that way_ (does not revert the table swap nor stops replication)
8 changes: 4 additions & 4 deletions doc/understanding-output.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

`gh-ost` attempts to be verbose to the point where you really know what it's doing, without completely spamming you.
You can control output levels:
- `--verbose`: common use. Useful output, not tons of it
- `--debug`: everything. Tons of output.
- `-verbose`: common use. Useful output, not tons of it
- `-debug`: everything. Tons of output.

Initial output lines may look like this:
```
Expand All @@ -16,7 +16,7 @@ Initial output lines may look like this:
2016-05-19 17:57:04 INFO Table found. Engine=InnoDB
2016-05-19 17:57:05 INFO As instructed, I'm issuing a SELECT COUNT(*) on the table. This may take a while
2016-05-19 17:57:11 INFO Exact number of rows via COUNT: 4466810
2016-05-19 17:57:11 INFO --test-on-replica given. Will not execute on master the.master:3306 but rather on replica 127.0.0.1:3306 itself
2016-05-19 17:57:11 INFO -test-on-replica given. Will not execute on master the.master:3306 but rather on replica 127.0.0.1:3306 itself
2016-05-19 17:57:11 INFO Master found to be 127.0.0.1:3306
2016-05-19 17:57:11 INFO connection validated on 127.0.0.1:3306
2016-05-19 17:57:11 INFO Registering replica at 127.0.0.1:3306
Expand Down Expand Up @@ -102,7 +102,7 @@ Notes:
- `Backlog: 100/100`: our buffer of `100` events is full; you may see this during or right after throttling (the binary logs keep filling up with relevant queries that are not being processed), or immediately following a high workload.
`gh-ost` will always prioritize binlog event processing (backlog) over row-copy; when next possible (throttling completes, in our example), `gh-ost` will drain the queue first, and only then proceed to resume row copy.
There is nothing wrong with seeing `100/100`; it just indicates we're behind at that point in time.
- `Copy: 31291200/43138418`, `Copy: 31389700/43138432`: this migration executed with `--exact-rowcount`. `gh-ost` continuously heuristically updates the total number of expected row copies as migration proceeds, hence the change from `43138418` to `43138432`
- `Copy: 31291200/43138418`, `Copy: 31389700/43138432`: this migration executed with `-exact-rowcount`. `gh-ost` continuously heuristically updates the total number of expected row copies as migration proceeds, hence the change from `43138418` to `43138432`
- `streamer: mysql-bin.006793:179473435` tells us which binary log entry is `gh-ost` processing at this time.

### Status hint
Expand Down
16 changes: 8 additions & 8 deletions go/cmd/gh-ost/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ func main() {

executeFlag := flag.Bool("execute", false, "actually execute the alter & migrate the table. Default is noop: do some tests and exit")
flag.BoolVar(&migrationContext.TestOnReplica, "test-on-replica", false, "Have the migration run on a replica, not on the master. At the end of migration replication is stopped, and tables are swapped and immediately swap-revert. Replication remains stopped and you can compare the two tables for building trust")
flag.BoolVar(&migrationContext.MigrateOnReplica, "migrate-on-replica", false, "Have the migration run on a replica, not on the master. This will do the full migration on the replica including cut-over (as opposed to --test-on-replica)")
flag.BoolVar(&migrationContext.MigrateOnReplica, "migrate-on-replica", false, "Have the migration run on a replica, not on the master. This will do the full migration on the replica including cut-over (as opposed to -test-on-replica)")

flag.BoolVar(&migrationContext.OkToDropTable, "ok-to-drop-table", false, "Shall the tool drop the old table at end of operation. DROPping tables can be a long locking operation, which is why I'm not doing it by default. I'm an online tool, yes?")
flag.BoolVar(&migrationContext.InitiallyDropOldTable, "initially-drop-old-table", false, "Drop a possibly existing OLD table (remains from a previous run?) before beginning operation. Default is to panic and abort if such table exists")
Expand All @@ -88,7 +88,7 @@ func main() {
flag.Int64Var(&migrationContext.ServeTCPPort, "serve-tcp-port", 0, "TCP port to serve on. Default: disabled")

maxLoad := flag.String("max-load", "", "Comma delimited status-name=threshold. e.g: 'Threads_running=100,Threads_connected=500'. When status exceeds threshold, app throttles writes")
criticalLoad := flag.String("critical-load", "", "Comma delimited status-name=threshold, same format as `--max-load`. When status exceeds threshold, app panics and quits")
criticalLoad := flag.String("critical-load", "", "Comma delimited status-name=threshold, same format as `-max-load`. When status exceeds threshold, app panics and quits")
quiet := flag.Bool("quiet", false, "quiet")
verbose := flag.Bool("verbose", false, "verbose")
debug := flag.Bool("debug", false, "debug mode (very verbose)")
Expand Down Expand Up @@ -127,23 +127,23 @@ func main() {
}

if migrationContext.DatabaseName == "" {
log.Fatalf("--database must be provided and database name must not be empty")
log.Fatalf("-database must be provided and database name must not be empty")
}
if migrationContext.OriginalTableName == "" {
log.Fatalf("--table must be provided and table name must not be empty")
log.Fatalf("-table must be provided and table name must not be empty")
}
if migrationContext.AlterStatement == "" {
log.Fatalf("--alter must be provided and statement must not be empty")
log.Fatalf("-alter must be provided and statement must not be empty")
}
migrationContext.Noop = !(*executeFlag)
if migrationContext.AllowedRunningOnMaster && migrationContext.TestOnReplica {
log.Fatalf("--allow-on-master and --test-on-replica are mutually exclusive")
log.Fatalf("-allow-on-master and -test-on-replica are mutually exclusive")
}
if migrationContext.AllowedRunningOnMaster && migrationContext.MigrateOnReplica {
log.Fatalf("--allow-on-master and --migrate-on-replica are mutually exclusive")
log.Fatalf("-allow-on-master and -migrate-on-replica are mutually exclusive")
}
if migrationContext.MigrateOnReplica && migrationContext.TestOnReplica {
log.Fatalf("--migrate-on-replica and --test-on-replica are mutually exclusive")
log.Fatalf("-migrate-on-replica and -test-on-replica are mutually exclusive")
}

switch *cutOver {
Expand Down
20 changes: 10 additions & 10 deletions go/logic/applier.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ const (
)

// Applier connects and writes the the applier-server, which is the server where migration
// happens. This is typically the master, but could be a replica when `--test-on-replica` or
// `--execute-on-replica` are given.
// happens. This is typically the master, but could be a replica when `-test-on-replica` or
// `-execute-on-replica` are given.
// Applier is the one to actually write row data and apply binlog events onto the ghost table.
// It is where the ghost & changelog tables get created. It is where the cut-over phase happens.
type Applier struct {
Expand Down Expand Up @@ -107,15 +107,15 @@ func (this *Applier) ValidateOrDropExistingTables() error {
}
}
if this.tableExists(this.migrationContext.GetGhostTableName()) {
return fmt.Errorf("Table %s already exists. Panicking. Use --initially-drop-ghost-table to force dropping it", sql.EscapeName(this.migrationContext.GetGhostTableName()))
return fmt.Errorf("Table %s already exists. Panicking. Use -initially-drop-ghost-table to force dropping it", sql.EscapeName(this.migrationContext.GetGhostTableName()))
}
if this.migrationContext.InitiallyDropOldTable {
if err := this.DropOldTable(); err != nil {
return err
}
}
if this.tableExists(this.migrationContext.GetOldTableName()) {
return fmt.Errorf("Table %s already exists. Panicking. Use --initially-drop-old-table to force dropping it", sql.EscapeName(this.migrationContext.GetOldTableName()))
return fmt.Errorf("Table %s already exists. Panicking. Use -initially-drop-old-table to force dropping it", sql.EscapeName(this.migrationContext.GetOldTableName()))
}

return nil
Expand Down Expand Up @@ -284,7 +284,7 @@ func (this *Applier) InitiateHeartbeat(heartbeatIntervalMilliseconds int64) {
}
}

// ExecuteThrottleQuery executes the `--throttle-query` and returns its results.
// ExecuteThrottleQuery executes the `-throttle-query` and returns its results.
func (this *Applier) ExecuteThrottleQuery() (int64, error) {
throttleQuery := this.migrationContext.GetThrottleQuery()

Expand Down Expand Up @@ -489,7 +489,7 @@ func (this *Applier) SwapTablesQuickAndBumpy() error {
}

// RenameTablesRollback renames back both table: original back to ghost,
// _old back to original. This is used by `--test-on-replica`
// _old back to original. This is used by `-test-on-replica`
func (this *Applier) RenameTablesRollback() (renameError error) {
// Restoring tables to original names.
// We prefer the single, atomic operation:
Expand Down Expand Up @@ -531,7 +531,7 @@ func (this *Applier) RenameTablesRollback() (renameError error) {
return log.Errore(renameError)
}

// StopSlaveIOThread is applicable with --test-on-replica; it stops the IO thread, duh.
// StopSlaveIOThread is applicable with -test-on-replica; it stops the IO thread, duh.
// We need to keep the SQL thread active so as to complete processing received events,
// and have them written to the binary log, so that we can then read them via streamer.
func (this *Applier) StopSlaveIOThread() error {
Expand All @@ -544,7 +544,7 @@ func (this *Applier) StopSlaveIOThread() error {
return nil
}

// StartSlaveSQLThread is applicable with --test-on-replica
// StartSlaveSQLThread is applicable with -test-on-replica
func (this *Applier) StopSlaveSQLThread() error {
query := `stop /* gh-ost */ slave sql_thread`
log.Infof("Verifying SQL thread is stopped")
Expand All @@ -555,7 +555,7 @@ func (this *Applier) StopSlaveSQLThread() error {
return nil
}

// StartSlaveSQLThread is applicable with --test-on-replica
// StartSlaveSQLThread is applicable with -test-on-replica
func (this *Applier) StartSlaveSQLThread() error {
query := `start /* gh-ost */ slave sql_thread`
log.Infof("Verifying SQL thread is running")
Expand All @@ -566,7 +566,7 @@ func (this *Applier) StartSlaveSQLThread() error {
return nil
}

// StopReplication is used by `--test-on-replica` and stops replication.
// StopReplication is used by `-test-on-replica` and stops replication.
func (this *Applier) StopReplication() error {
if err := this.StopSlaveIOThread(); err != nil {
return err
Expand Down
Loading

0 comments on commit bf9ecf3

Please sign in to comment.