Skip to content

Commit

Permalink
Doc changes: README, contributing, separate file for vttablet.
Browse files Browse the repository at this point in the history
  • Loading branch information
ryszard committed Jun 17, 2014
1 parent e63bf2d commit efab6f6
Show file tree
Hide file tree
Showing 3 changed files with 115 additions and 83 deletions.
65 changes: 22 additions & 43 deletions README.markdown
Original file line number Diff line number Diff line change
@@ -1,60 +1,39 @@
# Vitess

Vitess is a set of servers and tools meant to facilitate scaling of MySQL databases for the web.
It's currently used as a fundamental component of YouTube's MySQL infrastructure.
Vitess is a set of servers and tools meant to facilitate scaling of MySQL
databases for the web. It's currently used as a fundamental component of
YouTube's MySQL infrastructure, serving thousands of QPS (per server).

[sougou](https://github.com/sougou) presented Vitess at Fosdem '14 in the go devroom. [Here are the slides](https://github.com/youtube/vitess/blob/master/doc/Vitess2014.pdf?raw=true), and here is the [video](http://youtu.be/qATTTSg6zXk).
[sougou](https://github.com/sougou) presented Vitess at Fosdem '14 in the go
devroom. [Here are the
slides](https://github.com/youtube/vitess/blob/master/doc/Vitess2014.pdf?raw=true),
and here is the [video](http://youtu.be/qATTTSg6zXk).

Vitess is currently not ready for unsupervised use yet.
There's good deal of documentation missing.
Also, some functionality is under development and some APIs are still changing.
But if you feel adventurous, you're welcome to try it.
If you run into issues, please subscribe and post to [our mailing list](https://groups.google.com/forum/#!forum/vitess).
We'll do our best to help you.
## Trying it out

* [Vision](https://github.com/youtube/vitess/blob/master/doc/Vision.markdown)
* Concepts
Vitess is not entirely ready for unsupervised use yet. Some functionality is
still under development, APIs may change, and parts of the code are
undocumented. However, if you feel adventurous, you're more than welcome to try
it. We know that there are some rough edges, so please don't hesitate to reach
to us through [our mailing list](https://groups.google.com/forum/#!forum/vitess)
if you run into any issues. Warnings aside, please take a look at our [getting
started
doc](https://github.com/youtube/vitess/blob/master/doc/GettingStarted.markdown).

## Documentation

* [Vision](https://github.com/youtube/vitess/blob/master/doc/Vision.markdown)
* [General Concepts](https://github.com/youtube/vitess/blob/master/doc/Concepts.markdown)
* [Replication Graph](https://github.com/youtube/vitess/blob/master/doc/ReplicationGraph.markdown)
* [Serving graph](https://github.com/youtube/vitess/blob/master/doc/ServingGraph.markdown)
* [Tools](https://github.com/youtube/vitess/blob/master/doc/Tools.markdown)
* [Getting Started](https://github.com/youtube/vitess/blob/master/doc/GettingStarted.markdown)
* Operations
* [Tools](https://github.com/youtube/vitess/blob/master/doc/Tools.markdown)
* [Getting Started](https://github.com/youtube/vitess/blob/master/doc/GettingStarted.markdown)
* [Preparing for production](https://github.com/youtube/vitess/blob/master/doc/Production.markdown)
* [Reparenting](https://github.com/youtube/vitess/blob/master/doc/Reparenting.markdown)
* [Resharding](https://github.com/youtube/vitess/blob/master/doc/Resharding.markdown)
* [Schema management](https://github.com/youtube/vitess/blob/master/doc/SchemaManagement.markdown)
* [Zookeeper data](https://github.com/youtube/vitess/blob/master/doc/ZookeeperData.markdown)

### vttablet
TODO: Move this content to its own markdown.

Smart middleware sitting in front of MySQL and serving clients
requests.

* Connection pooling.
* SQL parser: Although very close, the vtocc SQL parser is not SQL-92
compliant. It has left out constructs that are deemed uncommon or
OLTP-unfriendly. It should, however, allow most queries used by a
well-behaved web application.
* Query rewrite and sanitation (adding limits, avoiding non-deterministic updates).
* Query consolidation: reuse the results of an in-flight query to any
subsequent requests that were received while the query was still
executing.
* Rowcache: the mysql buffer cache is optimized for range scans over
indices and tables. Unfortunately, it’s not good for random access
by primary key. The rowcache will instead maintain a row based cache
(using [memcached](http://memcached.org/) as its backend) and keep it
consistent by fielding all DMLs that could potentially affect them.
* Update stream: A server that streams the list of rows that are changing
in the database, which can be used as a mechanism to continuously export
the data to another data store.
* Integrated query killer for queries that take too long to return
data.
* Discard idle backend connections to avoid offline db errors.
* Transaction management: Ability to limit the number of concurrent
transactions and manage deadlines.

## License

Unless otherwise noted, the vitess source files are distributed
Expand Down
106 changes: 66 additions & 40 deletions doc/Contributing.markdown
Original file line number Diff line number Diff line change
@@ -1,24 +1,35 @@
# Contributing to Vitess
If you'd like to make simple contributions to Vitess,
we recommend that you fork the repository and submit pull requests.
If you'd like to make larger or ongoing changes,
you'll need to follow a similar set of processes and rules that the Vitess team follows.

If you'd like to make simple contributions to Vitess, we recommend that you fork
the repository and submit pull requests. If you'd like to make larger or ongoing
changes, you'll need to follow a similar set of processes and rules that the
Vitess team follows.

### Prerequisites
- [Install vitess](https://github.com/youtube/vitess/blob/master/doc/GettingStarted.markdown)
- The vitess team uses appspot for code reviews. You'll need to create an account at http://codereview.appspot.com.
- Fork the vitess repository, say https://github.com/myfork/vitess.
- Download [upload.py](https://code.google.com/p/rietveld/wiki/UploadPyUsage) and put it in your path.
- Add `$VTTOP/misc/git` to your path, or create a symlink from one of your paths to `$VTTOP/misc/git/createcl`.
- Subscribe to https://groups.google.com/forum/#!forum/vitess-issues.

### Single contributor
If you're going to be a sole contributor,
it means that you can use your fork as a push-only staging ground for submitting pull requests.
The assumption is that you'll never have to fetch from the fork.
If this is the case, all you have to do is configure your local repository to pull from youtube,
and push to myfork.
This can be achieved as follows:
- [Install vitess](https://github.com/youtube/vitess/blob/master/doc/GettingStarted.markdown)
- The vitess team uses appspot for code reviews. You'll need to create an account at http://codereview.appspot.com.
- Fork the vitess repository, say https://github.com/myfork/vitess.
- Download [upload.py](https://code.google.com/p/rietveld/wiki/UploadPyUsage) and put it in your path.



## Small changes

For small, well contained changes, just send us a
[pull request](https://help.github.com/articles/using-pull-requests).

## Bigger changes

If you are planning to make bigger changes or add serious features to Vitess, we
ask you to follow our code review process.

### Recommended Git flow: single contributor

Use your fork as a push-only staging ground for submitting pull requests. The
assumption is that you'll never have to fetch from the fork. If this is the
case, all you have to do is configure your local repository to pull from
youtube, and push to myfork. This can be achieved as follows:

```
~/...vitess> git remote -v
origin [email protected]:youtube/vitess.git (fetch)
Expand All @@ -29,17 +40,21 @@ origin [email protected]:youtube/vitess.git (fetch)
origin [email protected]:myfork/vitess (push)
```

The limitation of this configuration is that you can only pull from the youtube repository.
The `git pull` command will `fetch` from youtube/vitess and `merge` into your master branch.
The limitation of this configuration is that you can only pull from the youtube
repository. The `git pull` command will `fetch` from youtube/vitess and `merge`
into your master branch.

On the other hand, `git push` will push into your myfork/vitess remote.

The advantage of this workflow is that you don't have to worry about specifying where you're
pulling from or pushing to because the default settings *do the right thing*.
The advantage of this workflow is that you don't have to worry about specifying
where you're pulling from or pushing to because the default settings *do the
right thing*.

### Recommended Git flow: multiple contributors

### Multiple contributors
If more than one of you plan on contributing through a single fork,
then you'll need to follow a more elaborate scheme of setting up multiple remotes and manually managing merges:
If more than one of you plan on contributing through a single fork, then you'll
need to follow a more elaborate scheme of setting up multiple remotes and
manually managing merges:

```
~/...vitess> git remote -v
Expand All @@ -53,13 +68,15 @@ origin [email protected]:youtube/vitess.git (fetch)
origin [email protected]:youtube/vitess.git (push)
```

With this setup, commands like `git pull` and `git push` with default settings are not recommended.
You will be better off using `git fetch` and `git merge`, which let you micromanage your remote interactions.
For example, you'll need to `git push myfork` to explicitly push your changes to myfork.
With this setup, commands like `git pull` and `git push` with default settings
are not recommended. You will be better off using `git fetch` and `git merge`,
which let you micromanage your remote interactions. For example, you'll need to
`git push myfork` to explicitly push your changes to myfork.

### Changes and code reviews
We recommend that you make your changes in a separate branch.
Make sure you're on the master branch when you create it.

```
~/...vitess> git status
# On branch master
Expand All @@ -74,11 +91,18 @@ you can run the createcl tool, for example:
```
createcl -r alainjobart
```
This command will automatically run a diff of the current branch `newfeature` against `master`
and create an appspot code review with `alainjobart` as reviewer.
vitess-issues will be cc'd.
If necessary, createcl allows you to specify the exact versions to diff.
But we recommend that you don't use those.
This command will automatically run a diff of the current branch `newfeature`
against `master` and create an appspot code review with `alainjobart` as
reviewer. vitess-issues will be cc'd. If necessary, createcl allows you to
specify the exact versions to diff. (but we recommend that you don't use those).

After getting feedback about your code, you can update the code by calling

```
createcl -i 12345
```

with the actual id of your change instead of 12345.

During your feature development, you can fetch and merge new changes from the main youtube repository.
If you choose to do so, make sure you merge the changes to both the `master` and `newfeature` branches.
Expand All @@ -89,7 +113,7 @@ git pull
git checkout newfeature
git merge master
```
Once your change is approved, you have to push and submit it as a pull request:
Once your change is approved, push and submit it as a pull request:
```
git checkout master
git merge newfeature
Expand All @@ -103,10 +127,12 @@ github will cancel out changes you merged from youtube master, unless you resolv
If necessary, you can work on multiple branches at the same time.
When the time comes to submit, you just have to merge the branch onto `master` and push.

### More fancy github setups
As you can see above, the only requirement from the Vitess team is that you send your code reviews through appspot,
and then submit the same changes as a pull request.
## Other Git setups

As you can see above, the only requirement from the Vitess team is that you send
your code reviews through appspot, and then submit the same changes as a pull
request.

Our workflow recommendation is mainly to simplify your life.
If you prefer to use a different workflow,
you can choose to do so as long as you can figure out a way to meet the necessary requirements.
Our workflow recommendation is mainly to simplify your life. If you prefer to
use a different workflow, you can choose to do so as long as you can figure out
a way to meet the necessary requirements.
27 changes: 27 additions & 0 deletions doc/Vttablet.markdown
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# vttablet

Smart middleware sitting in front of MySQL and serving clients
requests.

* Connection pooling.
* SQL parser: Although very close, the vtocc SQL parser is not SQL-92
compliant. It has left out constructs that are deemed uncommon or
OLTP-unfriendly. It should, however, allow most queries used by a
well-behaved web application.
* Query rewrite and sanitation (adding limits, avoiding non-deterministic updates).
* Query consolidation: reuse the results of an in-flight query to any
subsequent requests that were received while the query was still
executing.
* Rowcache: the mysql buffer cache is optimized for range scans over
indices and tables. Unfortunately, it’s not good for random access
by primary key. The rowcache will instead maintain a row based cache
(using [memcached](http://memcached.org/) as its backend) and keep it
consistent by fielding all DMLs that could potentially affect them.
* Update stream: A server that streams the list of rows that are changing
in the database, which can be used as a mechanism to continuously export
the data to another data store.
* Integrated query killer for queries that take too long to return
data.
* Discard idle backend connections to avoid offline db errors.
* Transaction management: Ability to limit the number of concurrent
transactions and manage deadlines.

0 comments on commit efab6f6

Please sign in to comment.