forked from apache/druid
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
refactor extensions into core and contrib
- Loading branch information
Showing
271 changed files
with
652 additions
and
550 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| --- | ||
| layout: doc_page | ||
| --- | ||
|
|
||
| # Microsoft Azure | ||
|
|
||
| ## Deep Storage | ||
|
|
||
| [Microsoft Azure Storage](http://azure.microsoft.com/en-us/services/storage/) is another option for deep storage. This requires some additional druid configuration. | ||
|
|
||
| |Property|Possible Values|Description|Default| | ||
| |--------|---------------|-----------|-------| | ||
| |`druid.storage.type`|azure||Must be set.| | ||
| |`druid.azure.account`||Azure Storage account name.|Must be set.| | ||
| |`druid.azure.key`||Azure Storage account key.|Must be set.| | ||
| |`druid.azure.container`||Azure Storage container name.|Must be set.| | ||
| |`druid.azure.protocol`|http or https||https| | ||
| |`druid.azure.maxTries`||Number of tries before cancel an Azure operation.|3| | ||
|
|
||
| See [Azure Services](http://azure.microsoft.com/en-us/pricing/free-trial/) for more information. | ||
|
|
||
| ## Firehose | ||
|
|
||
| #### StaticAzureBlobStoreFirehose | ||
|
|
||
| This firehose ingests events, similar to the StaticS3Firehose, but from an Azure Blob Store. | ||
|
|
||
| Data is newline delimited, with one JSON object per line and parsed as per the `InputRowParser` configuration. | ||
|
|
||
| The storage account is shared with the one used for Azure deep storage functionality, but blobs can be in a different container. | ||
|
|
||
| As with the S3 blobstore, it is assumed to be gzipped if the extension ends in .gz | ||
|
|
||
| Sample spec: | ||
|
|
||
| ```json | ||
| "firehose" : { | ||
| "type" : "static-azure-blobstore", | ||
| "blobs": [ | ||
| { | ||
| "container": "container", | ||
| "path": "/path/to/your/file.json" | ||
| }, | ||
| { | ||
| "container": "anothercontainer", | ||
| "path": "/another/path.json" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| |property|description|default|required?| | ||
| |--------|-----------|-------|---------| | ||
| |type|This should be "static-azure-blobstore".|N/A|yes| | ||
| |blobs|JSON array of [Azure blobs](https://msdn.microsoft.com/en-us/library/azure/ee691964.aspx).|N/A|yes| | ||
|
|
||
| Azure Blobs: | ||
|
|
||
| |property|description|default|required?| | ||
| |--------|-----------|-------|---------| | ||
| |container|Name of the azure [container](https://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/#create-a-container)|N/A|yes| | ||
| |path|The path where data is located.|N/A|yes| |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| --- | ||
| layout: doc_page | ||
| --- | ||
|
|
||
| # Apache Cassandra | ||
|
|
||
| [Apache Cassandra](http://www.datastax.com/what-we-offer/products-services/datastax-enterprise/apache-cassandra) can also | ||
| be leveraged for deep storage. This requires some additional druid configuration as well as setting up the necessary | ||
| schema within a Cassandra keystore. |
65 changes: 65 additions & 0 deletions
65
docs/content/development/community-extensions/cloudfiles.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| --- | ||
| layout: doc_page | ||
| --- | ||
|
|
||
| # Rackspace Cloud Files | ||
|
|
||
| ## Deep Storage | ||
|
|
||
| [Rackspace Cloud Files](http://www.rackspace.com/cloud/files/) is another option for deep storage. This requires some additional druid configuration. | ||
|
|
||
| |Property|Possible Values|Description|Default| | ||
| |--------|---------------|-----------|-------| | ||
| |`druid.storage.type`|cloudfiles||Must be set.| | ||
| |`druid.storage.region`||Rackspace Cloud Files region.|Must be set.| | ||
| |`druid.storage.container`||Rackspace Cloud Files container name.|Must be set.| | ||
| |`druid.storage.basePath`||Rackspace Cloud Files base path to use in the container.|Must be set.| | ||
| |`druid.storage.operationMaxRetries`||Number of tries before cancel a Rackspace operation.|10| | ||
| |`druid.cloudfiles.userName`||Rackspace Cloud username|Must be set.| | ||
| |`druid.cloudfiles.apiKey`||Rackspace Cloud api key.|Must be set.| | ||
| |`druid.cloudfiles.provider`|rackspace-cloudfiles-us,rackspace-cloudfiles-uk|Name of the provider depending on the region.|Must be set.| | ||
| |`druid.cloudfiles.useServiceNet`|true,false|Whether to use the internal service net.|true| | ||
|
|
||
| ## Firehose | ||
|
|
||
| #### StaticCloudFilesFirehose | ||
|
|
||
| This firehose ingests events, similar to the StaticAzureBlobStoreFirehose, but from Rackspace's Cloud Files. | ||
|
|
||
| Data is newline delimited, with one JSON object per line and parsed as per the `InputRowParser` configuration. | ||
|
|
||
| The storage account is shared with the one used for Racksapce's Cloud Files deep storage functionality, but blobs can be in a different region and container. | ||
|
|
||
| As with the Azure blobstore, it is assumed to be gzipped if the extension ends in .gz | ||
|
|
||
| Sample spec: | ||
|
|
||
| ```json | ||
| "firehose" : { | ||
| "type" : "static-cloudfiles", | ||
| "blobs": [ | ||
| { | ||
| "region": "DFW" | ||
| "container": "container", | ||
| "path": "/path/to/your/file.json" | ||
| }, | ||
| { | ||
| "region": "ORD" | ||
| "container": "anothercontainer", | ||
| "path": "/another/path.json" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| |property|description|default|required?| | ||
| |--------|-----------|-------|---------| | ||
| |type|This should be "static-cloudfiles".|N/A|yes| | ||
| |blobs|JSON array of Cloud Files blobs.|N/A|yes| | ||
|
|
||
| Cloud Files Blobs: | ||
|
|
||
| |property|description|default|required?| | ||
| |--------|-----------|-------|---------| | ||
| |container|Name of the Cloud Files container|N/A|yes| | ||
| |path|The path where data is located.|N/A|yes| |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
22 changes: 13 additions & 9 deletions
22
...lopment/kafka-simple-consumer-firehose.md → ...ment/community-extensions/kafka-simple.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| --- | ||
| layout: doc_page | ||
| --- | ||
|
|
||
| # RabbitMQ | ||
|
|
||
| ## Firehose | ||
|
|
||
| #### RabbitMQFirehose | ||
|
|
||
| This firehose ingests events from a define rabbit-mq queue. | ||
|
|
||
| **Note:** Add **amqp-client-3.2.1.jar** to lib directory of druid to use this firehose. | ||
|
|
||
| A sample spec for rabbitmq firehose: | ||
|
|
||
| ```json | ||
| "firehose" : { | ||
| "type" : "rabbitmq", | ||
| "connection" : { | ||
| "host": "localhost", | ||
| "port": "5672", | ||
| "username": "test-dude", | ||
| "password": "test-word", | ||
| "virtualHost": "test-vhost", | ||
| "uri": "amqp://mqserver:1234/vhost" | ||
| }, | ||
| "config" : { | ||
| "exchange": "test-exchange", | ||
| "queue" : "druidtest", | ||
| "routingKey": "#", | ||
| "durable": "true", | ||
| "exclusive": "false", | ||
| "autoDelete": "false", | ||
| "maxRetries": "10", | ||
| "retryIntervalSeconds": "1", | ||
| "maxDurationSeconds": "300" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| |property|description|default|required?| | ||
| |--------|-----------|-------|---------| | ||
| |type|This should be "rabbitmq"|N/A|yes| | ||
| |host|The hostname of the RabbitMQ broker to connect to|localhost|no| | ||
| |port|The port number to connect to on the RabbitMQ broker|5672|no| | ||
| |username|The username to use to connect to RabbitMQ|guest|no| | ||
| |password|The password to use to connect to RabbitMQ|guest|no| | ||
| |virtualHost|The virtual host to connect to|/|no| | ||
| |uri|The URI string to use to connect to RabbitMQ| |no| | ||
| |exchange|The exchange to connect to| |yes| | ||
| |queue|The queue to connect to or create| |yes| | ||
| |routingKey|The routing key to use to bind the queue to the exchange| |yes| | ||
| |durable|Whether the queue should be durable|false|no| | ||
| |exclusive|Whether the queue should be exclusive|false|no| | ||
| |autoDelete|Whether the queue should auto-delete on disconnect|false|no| | ||
| |maxRetries|The max number of reconnection retry attempts| |yes| | ||
| |retryIntervalSeconds|The reconnection interval| |yes| | ||
| |maxDurationSeconds|The max duration of trying to reconnect| |yes| |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| layout: doc_page | ||
| --- | ||
|
|
||
| # RocketMQ | ||
|
|
||
| Original author: [https://github.com/lizhanhui](https://github.com/lizhanhui). |
Oops, something went wrong.