Skip to content

Commit

Permalink
chore: correct index cache config (#1484)
Browse files Browse the repository at this point in the history
  • Loading branch information
WenyXu authored Jan 21, 2025
1 parent b59794d commit 7cdba97
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 6 deletions.
16 changes: 13 additions & 3 deletions docs/user-guide/deployments/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -497,6 +497,13 @@ page_cache_size = "512MB"
sst_write_buffer_size = "8MB"
scan_parallelism = 0

[region_engine.mito.index]
aux_path = ""
staging_size = "2GB"
metadata_cache_size = "64MiB"
content_cache_size = "128MiB"
content_cache_page_size = "64KiB"

[region_engine.mito.inverted_index]
create_on_flush = "auto"
create_on_compaction = "auto"
Expand Down Expand Up @@ -534,15 +541,18 @@ Available options:
| `selector_result_cache_size` | String | `512MB` | Cache size for time series selector (e.g. `last_value()`). Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/8 of OS memory. |
| `sst_write_buffer_size` | String | `8MB` | Buffer size for SST writing. |
| `scan_parallelism` | Integer | `0` | Parallelism to scan a region (default: 1/4 of cpu cores).<br/>- `0`: using the default value (1/4 of cpu cores).<br/>- `1`: scan in current thread.<br/>- `n`: scan in parallelism n. |
| `index` | -- | -- | The options for index in Mito engine. |
| `index.aux_path` | String | `""` | Auxiliary directory path for the index in the filesystem. This path is used to store intermediate files for creating the index and staging files for searching the index. It defaults to `{data_home}/index_intermediate`. The default name for this directory is `index_intermediate` for backward compatibility. This path contains two subdirectories: `__intm` for storing intermediate files used during index creation, and `staging` for storing staging files used during index searching. |
| `index.staging_size` | String | `2GB` | The maximum capacity of the staging directory. |
| `index.metadata_cache_size` | String | `64MiB` | Cache size for index metadata. |
| `index.content_cache_size` | String | `128MiB` | Cache size for index content. |
| `index.content_cache_page_size` | String | `64KiB` | Page size for index content cache. |
| `inverted_index` | -- | -- | The options for inverted index in Mito engine. |
| `inverted_index.create_on_flush` | String | `auto` | Whether to create the index on flush.<br/>- `auto`: automatically<br/>- `disable`: never |
| `inverted_index.create_on_compaction` | String | `auto` | Whether to create the index on compaction.<br/>- `auto`: automatically<br/>- `disable`: never |
| `inverted_index.apply_on_query` | String | `auto` | Whether to apply the index on query<br/>- `auto`: automatically<br/>- `disable`: never |
| `inverted_index.mem_threshold_on_create` | String | `64M` | Memory threshold for performing an external sort during index creation.<br/>Setting to empty will disable external sorting, forcing all sorting operations to happen in memory. |
| `inverted_index.intermediate_path` | String | `""` | File system path to store intermediate files for external sorting (default `{data_home}/index_intermediate`). |
| `inverted_index.metadata_cache_size` | String | `64MiB` | Cache size for inverted index metadata. |
| `inverted_index.content_cache_size` | String | `128MiB` | Cache size for inverted index content. |
| `inverted_index.content_cache_page_size`| String | `64KiB` | Page size for inverted index content cache. Inverted index content will be read and cached in page size. Adjust this value to change the granularity of cache and optimize the cache hit rate. |
| `memtable.type` | String | `time_series` | Memtable type.<br/>- `time_series`: time-series memtable<br/>- `partition_tree`: partition tree memtable (experimental) |
| `memtable.index_max_keys_per_shard` | Integer | `8192` | The max number of keys in one shard.<br/>Only available for `partition_tree` memtable. |
| `memtable.data_freeze_threshold` | Integer | `32768` | The max rows of data inside the actively writing buffer in one shard.<br/>Only available for `partition_tree` memtable. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -487,6 +487,13 @@ page_cache_size = "512MB"
sst_write_buffer_size = "8MB"
scan_parallelism = 0

[region_engine.mito.index]
aux_path = ""
staging_size = "2GB"
metadata_cache_size = "64MiB"
content_cache_size = "128MiB"
content_cache_page_size = "64KiB"

[region_engine.mito.inverted_index]
create_on_flush = "auto"
create_on_compaction = "auto"
Expand Down Expand Up @@ -524,14 +531,17 @@ fork_dictionary_bytes = "1GiB"
| `selector_result_cache_size` | 字符串 | `512MB` | `last_value()` 等时间线检索结果的缓存。设为 0 可关闭该缓存<br/>默认为内存的 1/16,不超过 512MB |
| `sst_write_buffer_size` | 字符串 | `8MB` | SST 的写缓存大小 |
| `scan_parallelism` | 整数 | `0` | 扫描并发度 (默认 1/4 CPU 核数)<br/>- `0`: 使用默认值 (1/4 CPU 核数)<br/>- `1`: 单线程扫描<br/>- `n`: 按并行度 n 扫描 |
| `index` | -- | -- | Mito 引擎中索引的选项。 |
| `index.aux_path` | 字符串 | `""` | 文件系统中索引的辅助目录路径,用于存储创建索引的中间文件和搜索索引的暂存文件,默认为 `{data_home}/index_intermediate`。为了向后兼容,该目录的默认名称为 `index_intermediate`。此路径包含两个子目录:- `__intm`: 用于存储创建索引时使用的中间文件。- `staging`: 用于存储搜索索引时使用的暂存文件。 |
| `index.staging_size` | 字符串 | `2GB` | 暂存目录的最大容量。 |
| `index.metadata_cache_size` | 字符串 | `64MiB` | 索引元数据的缓存大小。 |
| `index.content_cache_size` | 字符串 | `128MiB` | 索引内容的缓存大小。 |
| `index.content_cache_page_size` | 字符串 | `64KiB` | 倒排索引内容缓存的页大小。 |
| `inverted_index.create_on_flush` | 字符串 | `auto` | 是否在 flush 时构建索引<br/>- `auto`: 自动<br/>- `disable`: 从不 |
| `inverted_index.create_on_compaction` | 字符串 | `auto` | 是否在 compaction 时构建索引<br/>- `auto`: 自动<br/>- `disable`: 从不 |
| `inverted_index.apply_on_query` | 字符串 | `auto` | 是否在查询时使用索引<br/>- `auto`: 自动<br/>- `disable`: 从不 |
| `inverted_index.mem_threshold_on_create` | 字符串 | `64M` | 创建索引时如果超过该内存阈值则改为使用外部排序<br/>设置为空会关闭外排,在内存中完成所有排序 |
| `inverted_index.intermediate_path` | 字符串 | `""` | 存放外排临时文件的路径 (默认 `{data_home}/index_intermediate`). |
| `inverted_index.metadata_cache_size` | 字符串 | `64MiB` | 倒排索引元数据缓存大小 |
| `inverted_index.content_cache_size` | 字符串 | `128MiB` | 倒排索引文件内容缓存大小 |
| `inverted_index.content_cache_page_size` | 字符串 | `64KiB` | 倒排索引文件内容缓存页大小。倒排索引文件内容以页为单位进行读取和缓存,该配置项用于调整读取和缓存的粒度,优化缓存命中率。 |
| `memtable.type` | 字符串 | `time_series` | Memtable type.<br/>- `time_series`: time-series memtable<br/>- `partition_tree`: partition tree memtable (实验性功能) |
| `memtable.index_max_keys_per_shard` | 整数 | `8192` | 一个 shard 内的主键数<br/>只对 `partition_tree` memtable 生效 |
| `memtable.data_freeze_threshold` | 整数 | `32768` | 一个 shard 内写缓存可容纳的最大行数<br/>只对 `partition_tree` memtable 生效 |
Expand Down

0 comments on commit 7cdba97

Please sign in to comment.