Skip to content

Commit

Permalink
Cleanup the base lookup cluster wide config docs (apache#3061)
Browse files Browse the repository at this point in the history 
* Cleanup the base lookup cluster wide config docs

* Add better examples in lookups-cached-global.md

* Use actual valid stock lookups

* Fixed maps with :

* Add mix of lookups

* Better examples in extension

* Remove unneeded namespace requirement

* Add extra line space

* Add link to lookup tiers

* Renamed header
  • Loading branch information
@drcrallen @fjy
drcrallen authored and fjy committed Jun 7, 2016
1 parent e6b32c2 commit fa41a64
Show file tree
Hide file tree
Showing 2 changed files with 108 additions and 64 deletions.
56 changes: 53 additions & 3 deletions docs/content/development/extensions-core/lookups-cached-global.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ Cached namespace lookups can be specified as part of the [cluster wide config fo
"type": "cachedNamespace",
"extractionNamespace": {
"type": "uri",
"namespace": "some_uri_lookup",
"uri": "file:/tmp/prefix/",
"namespaceParseSpec": {
"format": "csv",
Expand All @@ -48,7 +47,6 @@ Cached namespace lookups can be specified as part of the [cluster wide config fo
"type": "cachedNamespace",
"extractionNamespace": {
"type": "jdbc",
"namespace": "some_jdbc_lookup",
"connectorConfig": {
"createTables": true,
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
Expand Down Expand Up @@ -78,6 +76,58 @@ If `firstCacheTimeout` is set to a non-zero value, it should be less than `druid
Proper functionality of Namespaced lookups requires the following extension to be loaded on the broker, peon, and historical nodes:
`druid-lookups-cached-global`

## Example configuration

In a simple case where only one [tier](../../querying/lookups.html#dynamic-configuration) exists (`realtime_customer2`) with one `cachedNamespace` lookup called `country_code`, the resulting configuration json looks similar to the following:

```json
{
"realtime_customer2": {
"country_code": {
"type": "cachedNamespace",
"extractionNamespace": {
"type": "jdbc",
"connectorConfig": {
"createTables": true,
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
"user": "druid",
"password": "diurd"
},
"table": "lookupTable",
"keyColumn": "country_id",
"valueColumn": "country_name",
"tsColumn": "timeColumn"
},
"firstCacheTimeout": 120000,
"injective":true
}
}
}
```

Where the coordinator endpoint `/druid/coordinator/v1/lookups/realtime_customer2/country_code` should return

```json
{
"type": "cachedNamespace",
"extractionNamespace": {
"type": "jdbc",
"connectorConfig": {
"createTables": true,
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
"user": "druid",
"password": "diurd"
},
"table": "lookupTable",
"keyColumn": "country_id",
"valueColumn": "country_name",
"tsColumn": "timeColumn"
},
"firstCacheTimeout": 120000,
"injective":true
}
```

## Cache Settings

Lookups are cached locally on historical nodes. The following are settings used by the nodes which service queries when
Expand Down Expand Up @@ -126,9 +176,9 @@ The remapping values for each namespaced lookup can be specified by a json objec
"pollPeriod":"PT5M"
}
```

|Property|Description|Required|Default|
|--------|-----------|--------|-------|
|`namespace`|The namespace to define|Yes||
|`pollPeriod`|Period between polling for updates|No|0 (only once)|
|`uri`|URI for the file of interest|No|Use `uriPrefix`|
|`uriPrefix`|A URI which specifies a directory (or other searchable resource) in which to search for files|No|Use `uri`|
Expand Down
116 changes: 55 additions & 61 deletions docs/content/querying/lookups.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ Lookups can be updated in bulk by posting a JSON object to `/druid/coordinator/v
{
"tierName": {
"lookupExtractorFactoryName": {
"type": "someExtractorFactoryType",
"someExtractorField": "someExtractorValue"
}
}
Expand All @@ -91,55 +92,54 @@ So a config might look something like:
{
"__default": {
"country_code": {
"type": "simple_json",
"uri": "http://some.host.com/codes.json"
"type": "map",
"map": {"77483": "United States"}
},
"site_id": {
"type": "confidential_jdbc",
"auth": "/etc/jdbc.internal",
"table": "sites",
"key": "site_id",
"value": "site_name"
"type": "cachedNamespace",
"extractionNamespace": {
"type": "jdbc",
"connectorConfig": {
"createTables": true,
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
"user": "druid",
"password": "diurd"
},
"table": "lookupTable",
"keyColumn": "country_id",
"valueColumn": "country_name",
"tsColumn": "timeColumn"
},
"firstCacheTimeout": 120000,
"injective":true
},
"site_id_customer1": {
"type": "confidential_jdbc",
"auth": "/etc/jdbc.customer1",
"table": "sites",
"key": "site_id",
"value": "site_name"
"type": "map",
"map": {"847632": "Internal Use Only"}
},
"site_id_customer2": {
"type": "confidential_jdbc",
"auth": "/etc/jdbc.customer2",
"table": "sites",
"key": "site_id",
"value": "site_name"
"type": "map",
"map": {"AHF77": "Home"}
}
},
"realtime_customer1": {
"country_code": {
"type": "simple_json",
"uri": "http://some.host.com/codes.json"
"type": "map",
"map": {"77483": "United States"}
},
"site_id_customer1": {
"type": "confidential_jdbc",
"auth": "/etc/jdbc.customer1",
"table": "sites",
"key": "site_id",
"value": "site_name"
"type": "map",
"map": {"847632": "Internal Use Only"}
}
},
"realtime_customer2": {
"country_code": {
"type": "simple_json",
"uri": "http://some.host.com/codes.json"
"type": "map",
"map": {"77483": "United States"}
},
"site_id_customer2": {
"type": "confidential_jdbc",
"auth": "/etc/jdbc.customer2",
"table": "sites",
"key": "site_id",
"value": "site_name"
"type": "map",
"map": {"AHF77": "Home"}
}
}
}
Expand All @@ -154,11 +154,8 @@ For example, a post to `/druid/coordinator/v1/lookups/realtime_customer1/site_id

```json
{
"type": "confidential_jdbc",
"auth": "/etc/jdbc.customer1",
"table": "sites_updated",
"key": "site_id",
"value": "site_name"
"type": "map",
"map": {"847632": "Internal Use Only"}
}
```

Expand All @@ -171,11 +168,8 @@ Using the prior example, a `GET` to `/druid/coordinator/v1/lookups/realtime_cust

```json
{
"type": "confidential_jdbc",
"auth": "/etc/jdbc.customer2",
"table": "sites",
"key": "site_id",
"value": "site_name"
"type": "map",
"map": {"AHF77": "Home"}
}
```

Expand Down Expand Up @@ -205,10 +199,10 @@ The return value will be a json map of the lookups to their extractor factories.
```json

{
"some_lookup_name": {
"type": "simple_json",
"uri": "http://some.host.com/codes.json"
}
"some_lookup_name": {
"type": "map",
"map": {"77483": "United States"}
}
}

```
Expand All @@ -220,8 +214,8 @@ The return value will be the json representation of the factory.

```json
{
"type": "simple_json",
"uri": "http://some.host.com/codes.json"
"type": "map",
"map": {"77483", "United States"}
}
```

Expand All @@ -232,8 +226,8 @@ The return value will be a JSON map in the following format:

```json
{
"status": "accepted",
"failedUpdates": {}
"status": "accepted",
"failedUpdates": {}
}

```
Expand All @@ -242,13 +236,13 @@ If a lookup cannot be started, or is left in an undefined state, the lookup in e

```json
{
"status": "accepted",
"failedUpdates": {
"country_code": {
"type": "simple_json",
"uri": "http://some.host.com/codes.json"
}
"status": "accepted",
"failedUpdates": {
"country_code": {
"type": "map",
"map": {"77483": "United States"}
}
}
}

```
Expand All @@ -263,8 +257,8 @@ If `some_lookup_name` is desired to have the LookupExtractorFactory definition o

```json
{
"type": "simple_json",
"uri": "http://some.host.com/codes.json"
"type": "map",
"map": {"77483": "United States"}
}
```

Expand All @@ -273,10 +267,10 @@ Then a post to `/druid/listen/v1/lookups/some_lookup_name` will behave the same
```json

{
"some_lookup_name": {
"type": "simple_json",
"uri": "http://some.host.com/codes.json"
}
"some_lookup_name": {
"type": "map",
"map": {"77483": "United States"}
}
}

```
Expand Down

0 comments on commit fa41a64

Please sign in to comment.