Skip to content

Commit

Permalink
do some editing of the instructions for using mysql for metadata
Browse files Browse the repository at this point in the history
  • Loading branch information
rasahner committed Jan 21, 2016
1 parent dcb7830 commit c936870
Show file tree
Hide file tree
Showing 9 changed files with 58 additions and 39 deletions.
2 changes: 1 addition & 1 deletion docs/content/configuration/production-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ To setup a local Druid cluster, see [Simple Cluster Configuration](../configurat

```
# Extensions
druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions", "io.druid.extensions:druid-histogram", "io.druid.extensions:mysql-metadata-storage"]
druid.extensions.loadList=["druid-s3-extensions", "druid-histogram", "mysql-metadata-storage"]
# Zookeeper
druid.zk.service.host=#{ZK_IPs}
Expand Down
2 changes: 1 addition & 1 deletion docs/content/configuration/simple-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This simple Druid cluster configuration can be used for initially experimenting

```
# Extensions
-Ddruid.extensions.coordinates=["io.druid.extensions:druid-kafka-eight"]
-Ddruid.extensions.loadList=["druid-kafka-eight"]
# Zookeeper (defaults to localhost)
Expand Down
2 changes: 1 addition & 1 deletion docs/content/dependencies/cassandra-deep-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ First create the schema above. I use a new keyspace called `druid` for this purp
Then, add the following to your historical and realtime runtime properties files to enable a Cassandra backend.

```properties
druid.extensions.coordinates=["io.druid.extensions:druid-cassandra-storage:<druid version>"]
druid.extensions.loadList=["druid-cassandra-storage"]
druid.storage.type=c*
druid.storage.host=localhost:9160
druid.storage.keyspace=druid
Expand Down
57 changes: 38 additions & 19 deletions docs/content/dependencies/metadata-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,28 @@ a number of tables used for various purposes described below.

The following metadata storage engines are supported:

* Derby (default - only works if you have all processes running on the same node)
* MySQL (io.druid.extensions:mysql-metadata-storage)
* PostgreSQL (io.druid.extensions:postgresql-metadata-storage)

To choose a metadata storage,

1. Make sure Druid can pick up the extension files from either classpath or
extensions directory, see [Including Extensions](../operations/including-extensions.html) for more information.

2. set the `druid.extensions` configuration to include the extension for the
metadata storage you plan to use. See below.

* Derby (default, but not suitable for production)
* MySQL
* PostgreSQL

Even though Derby is the default, it works only if you have all Druid
processes running on the same host, and should be used only for experimentation.
For production, MySQL or PostgreSQL should be used.

To choose the metadata storage type, set `druid.metadata.storage.type` to
`mysql`, `postgres` or `derby`.
Set other `druid.metadata.storage` configuration
keywords as shown below to give Druid information about how to connect to
the database.

As discussed in [Including Extensions](../operations/including-extensions.html),
there are two ways for giving Druid the extension files it needs for the
database you are using.
The first is to put the extension files in the classpath. The second is to
put the extension files in a subdirectory of
`druid.extensions.directory` (by default `extensions` under the Druid working directory) and list the subdirectory name in
`druid.extensions.loadList`. The example properties below show the second
way.

## Setting up MySQL

Expand Down Expand Up @@ -57,20 +67,20 @@ metadata storage you plan to use. See below.
3. Configure your Druid metadata storage extension:

Add the following parameters to your Druid configuration, replacing `<host>`
with the hostname of the database.
with the location (host name and port) of the database.

```properties
druid.extensions.loadList=["mysql-metadata-storage"]
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://<host>/druid_test
druid.metadata.storage.connector.connectURI=jdbc:mysql://<host>/druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=diurd
```

Note: metadata storage extension is not packaged within the main Druid tarball, it is
Note: the metadata storage extension is not packaged within the main Druid tarball; it is
packaged in a separate tarball that can be downloaded from [here](http://druid.io/downloads.html).
However, you can always get it using [pull-deps](../pull-deps.html), or you can even build
it from source code, see [Build from Source](../development/build.html)
You can also get it using [pull-deps](../pull-deps.html), or you can build
it from source code; see [Build from Source](../development/build.html).

## Setting up PostgreSQL

Expand Down Expand Up @@ -104,16 +114,25 @@ metadata storage you plan to use. See below.
3. Configure your Druid metadata storage extension:

Add the following parameters to your Druid configuration, replacing `<host>`
with the hostname of the database.
with the location (host name and port) of the database.

```properties
druid.extensions.loadList=["postgresql-metadata-storage"]
druid.metadata.storage.type=postgresql
druid.metadata.storage.connector.connectURI=jdbc:postgresql://<host>/druid_test
druid.metadata.storage.connector.connectURI=jdbc:postgresql://<host>/druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=diurd
```

## Using derby

Add the following to your Druid configuration.

```properties
druid.metadata.storage.type=derby
druid.metadata.storage.connector.connectURI=jdbc:derby://localhost:1527//home/y/var/druid_state/derby;create=true
```

## Metadata Storage Tables

### Segments Table
Expand Down
8 changes: 4 additions & 4 deletions docs/content/development/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ layout: doc_page

### Build from Source

The other way to setup Druid is from source via git.
Druid can be set up by building from source via git.

Builing Druid requires the following:
Building Druid requires the following:
- [JDK 7](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html)
or [JDK 8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)
- [Maven version 3.x](http://maven.apache.org/download.cgi)
Expand All @@ -22,8 +22,8 @@ mvn clean install
This will compile the project and create the Druid binary distribution tar under
`distribution/target/druid-VERSION-bin.tar.gz`.

This will also create a tarball that contains `mysql-metadata-storage` extension under
`distribution/target/mysql-metadata-storage-bin.tar.gz`. If you want Druid to load `mysql-metadata-storage`, you can first untar `druid-VERSION-bin.tar.gz`, then go to ```druid-<version>/extensions```, untar `mysql-metadata-storage-bin.tar.gz` there. Now just specifiy `mysql-metadata-storage` in `druid.extensions.loadList` so that Druid will pick it up. See [Including Extensions](../operations/including-extensions.html) for more infomation.
This will also create `distribution/target/mysql-metadata-storage-bin.tar.gz`,
which is a tarball that contains the `mysql-metadata-storage` extension.

You can find the example executables in the examples/bin directory:

Expand Down
6 changes: 3 additions & 3 deletions docs/content/development/experimental.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ layout: doc_page
Experimental features are features we have developed but have not fully tested in a production environment. If you choose to try them out, there will likely be edge cases that we have not covered. We would love feedback on any of these features, whether they are bug reports, suggestions for improvement, or letting us know they work as intended.


To enable experimental features, include their artifacts in the configuration runtime.properties file. Eg-
To enable experimental features, include their artifacts in the configuration runtime.properties file, e.g.,

```
druid.extensions.coordinates=["io.druid.extensions:druid-histogram:{VERSION}"]
druid.extensions.loadList=["druid-histogram"]
```

The configuration for all the indexer and query nodes need to be updated with this.
The configuration files for all the indexer and query nodes need to be updated with this.
10 changes: 5 additions & 5 deletions docs/content/operations/including-extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ If you don't want to fiddle with classpath, you can create an extension director

To let Druid load your extensions, follow the steps below

1) Specify `druid.extensions.directory` (root directory for normal Druid extensions). If you don' specify it, Druid will use their default value, see [Configuration](../configuration/index.html).
1) Specify `druid.extensions.directory` (root directory for normal Druid extensions). If you don't specify it, Druid will use the default value, see [Configuration](../configuration/index.html).

2) Prepare normal extension directories under root extension directory. Under the root extension directory, you should create sub-directories for each extension you might want to load. Inside each sub-directory, you can put extension related files in it. (If you don't want to manually setup the extension directory, Druid also provides a [pull-deps](../pull-deps.html) tool that can help you generate these directories automatically)
2) Under the root extension directory, create sub-directories for each extension you might want to load. Inside each sub-directory, put extension-related files. (If you don't want to manually set up the extension directory, you can use Druid's [pull-deps](../pull-deps.html) tool to help you generate these directories automatically.)

Example:

Expand Down Expand Up @@ -63,8 +63,8 @@ extensions/

As you can see, under ```extensions``` there are three sub-directories ```druid-examples```, ```druid-kafka-eight``` and ```mysql-metadata-storage```, each sub-directory denotes an extension that Druid might load.

3) Tell Druid which extensions to load. Now you have prepared your extension directories, if you want Druid to load a specific list of extensions under root extension directory, you need to specify `druid.extensions.loadList`. Using the example above, if you want Druid to load ```druid-kafka-eight``` and ```mysql-metadata-storage```, you can specify `druid.extensions.loadList=["druid-kafka-eight", "mysql-metadata-storage"]`.
3) To have Druid load a specific list of extensions present under the root extension directory, set `druid.extensions.loadList` to the list of extensions to load. Using the example above, if you want Druid to load ```druid-kafka-eight``` and ```mysql-metadata-storage```, you can specify `druid.extensions.loadList=["druid-kafka-eight", "mysql-metadata-storage"]`.

If you specify `druid.extensions.loadList=[]`, Druid won't load any extension from file system.
If you specify `druid.extensions.loadList=[]`, Druid won't load any extensions from the file system.

If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under root extension directory.
If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under the root extension directory.
6 changes: 3 additions & 3 deletions docs/content/tutorials/tutorial-a-first-look-at-druid.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ About the data

The data source we'll be working with is Wikipedia edits. Each time an edit is made in Wikipedia, an event gets pushed to an IRC channel associated with the language of the Wikipedia page. We scrape IRC channels for several different languages and load this data into Druid.

Each event has a timestamp indicating the time of the edit (in UTC time), a list of dimensions indicating various metadata about the event (such as information about the user editing the page and where the user is a bot), and a list of metrics associated with the event (such as the number of characters added and deleted).
Each event has a timestamp indicating the time of the edit (in UTC time), a list of dimensions indicating various metadata about the event (such as information about the user editing the page and whether the user is a bot), and a list of metrics associated with the event (such as the number of characters added and deleted).

Specifically. the data schema looks like so:
Specifically. the data schema looks like this:

Dimensions (things to filter on):

Expand Down Expand Up @@ -53,7 +53,7 @@ We've built a tarball that contains everything you'll need. You'll find it [here

### Build From Source

Follow the [Build From Source](../development/build.html) guide to build from source. Then grab the tarball from services/target/druid-<version>-bin.tar.gz.
Follow the [Build From Source](../development/build.html) guide to build from source. Then grab the tarball from distribution/target/druid-<version>-bin.tar.gz.

### Unpack the Tarball

Expand Down
4 changes: 2 additions & 2 deletions docs/content/tutorials/tutorial-the-druid-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ In this tutorial, we will set up other types of Druid nodes and external depende
## Downloading Druid
If you followed the first tutorial, you should already have Druid downloaded. If not, let's go back and do that first.

You can download the latest version of druid [here](http://druid.io/downloads.html). You can also [Build From Source](../development/build.html) and grab the tarball from services/target/druid-<version>-bin.tar.gz.
You can download the latest version of druid [here](http://druid.io/downloads.html). You can also [Build From Source](../development/build.html) and grab the tarball from distribution/target/druid-<version>-bin.tar.gz.

Either way, once you have the tarball, untar the contents within by issuing:

Expand Down Expand Up @@ -95,7 +95,7 @@ In the directory, there should be a `common.runtime.properties` file with the fo

```
# Extensions
druid.extensions.coordinates=["io.druid.extensions:druid-examples","io.druid.extensions:druid-kafka-eight"]
druid.extensions.loadList=["druid-examples","druid-kafka-eight"]
# Zookeeper
druid.zk.service.host=localhost
Expand Down

0 comments on commit c936870

Please sign in to comment.