Skip to content

Commit

Permalink
docs: hnsw indexes
Browse files Browse the repository at this point in the history
  • Loading branch information
gregnr committed Sep 6, 2023
1 parent cfaf474 commit 7042eb0
Show file tree
Hide file tree
Showing 13 changed files with 207 additions and 49 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -977,8 +977,8 @@ export const ai: NavMenuConstant = {
url: undefined,
items: [
{ name: 'Managing collections', url: '/guides/ai/managing-collections' },
{ name: 'Managing indexes', url: '/guides/ai/managing-indexes' },
{ name: 'Vector columns', url: '/guides/ai/vector-columns' },
{ name: 'Vector indexes', url: '/guides/ai/vector-indexes' },
{ name: 'Engineering for scale', url: '/guides/ai/engineering-for-scale' },
{ name: 'Choosing Compute Add-on', url: '/guides/ai/choosing-compute-addon' },
{ name: 'Going to Production', url: '/guides/ai/going-to-prod' },
Expand Down
2 changes: 1 addition & 1 deletion apps/docs/pages/guides/ai/managing-collections.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ The time required to create an index grows with the number of records and size o

</Admonition>

For an in-depth guide on vector indexes, see [Managing indexes](/docs/guides/ai/managing-indexes).
For an in-depth guide on vector indexes, see [Vector indexes](/docs/guides/ai/vector-indexes).

### Query

Expand Down
2 changes: 1 addition & 1 deletion apps/docs/pages/guides/ai/vector-columns.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ Vectors and embeddings can be used for much more than search. Learn more about e

### Indexes

Once your vector table starts to grow, you will likely want to add an index to speed up queries. See [Managing indexes](/docs/guides/ai/managing-indexes) to learn how vector indexes work and how to create them.
Once your vector table starts to grow, you will likely want to add an index to speed up queries. See [Vector indexes](/docs/guides/ai/vector-indexes) to learn how vector indexes work and how to create them.

export const Page = ({ children }) => <Layout meta={meta} children={children} />

Expand Down
43 changes: 43 additions & 0 deletions apps/docs/pages/guides/ai/vector-indexes.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
import Layout from '~/layouts/DefaultGuideLayout'

export const meta = {
id: 'ai-vector-indexes',
title: 'Vector indexes',
description: 'Understanding vector indexes',
sidebar_label: 'Vector indexes',
}

Once your vector table starts to grow, you will likely want to add an index to speed up queries. Without indexes, you'll be performing a sequential scan which can be a resource-intensive operation when you have many records.

## Choosing an index

Today `pgvector` supports two types of indexes:

- [HNSW](/docs/guides/ai/vector-indexes/hnsw-indexes)
- [IVFFlat](/docs/guides/ai/vector-indexes/ivf-indexes)

In general we recommend using [HNSW](/docs/guides/ai/vector-indexes/hnsw-indexes) because of its [performance gain](/blog/increase-performance-pgvector-hnsw#hnsw-performance-1536-dimensions) and [robustness against changing data](/docs/guides/ai/vector-indexes/hnsw-indexes#when-should-you-create-hnsw-indexes).

## Distance operators

The type of index required depends on the distance operator you are using. `pgvector` includes 3 distance operators:

| Operator | Description | [**Operator class**](https://www.postgresql.org/docs/current/sql-createopclass.html) |
| -------- | ---------------------- | ------------------------------------------------------------------------------------ |
| `<->` | Euclidean distance | `vector_l2_ops` |
| `<#>` | negative inner product | `vector_ip_ops` |
| `<=>` | cosine distance | `vector_cosine_ops` |

Use the following SQL commands to create an index for the operator(s) used in your queries.

Currently vectors with up to 2,000 dimensions can be indexed.

If you are using the `vecs` Python library, follow the instructions in [Managing collections](/docs/guides/ai/managing-collections#create-an-index) to create indexes.

## Resources

Read more about indexing on `pgvector`'s [GitHub page](https://github.com/pgvector/pgvector#indexing).

export const Page = ({ children }) => <Layout meta={meta} children={children} />

export default Page
101 changes: 101 additions & 0 deletions apps/docs/pages/guides/ai/vector-indexes/hnsw-indexes.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
import Layout from '~/layouts/DefaultGuideLayout'

export const meta = {
id: 'ai-hnsw-indexes',
title: 'HNSW indexes',
description: 'Understanding HNSW indexes in pgvector',
sidebar_label: 'HNSW indexes',
}

HNSW is a type of vector index for approximate nearest neighbor search, used in high-dimensional spaces like those found in embeddings.

## Usage

The way you create an HNSW index depends on the distance operator you are using. `pgvector` includes 3 distance operators:

| Operator | Description | [**Operator class**](https://www.postgresql.org/docs/current/sql-createopclass.html) |
| -------- | ---------------------- | ------------------------------------------------------------------------------------ |
| `<->` | Euclidean distance | `vector_l2_ops` |
| `<#>` | negative inner product | `vector_ip_ops` |
| `<=>` | cosine distance | `vector_cosine_ops` |

Use the following SQL commands to create an HNSW index for the operator(s) used in your queries.

### Euclidean L2 distance (`vector_l2_ops`)

```sql
create index on items using hnsw (column_name vector_l2_ops);
```

### Inner product (`vector_ip_ops`)

```sql
create index on items using hnsw (column_name vector_ip_ops);
```

### Cosine distance (`vector_cosine_ops`)

```sql
create index on items using hnsw (column_name vector_cosine_ops);
```

Currently vectors with up to 2,000 dimensions can be indexed.

If you are using the `vecs` Python library, follow the instructions in [Managing collections](/docs/guides/ai/managing-collections#create-an-index) to create indexes.

## How does HNSW work?

HNSW uses proximity graphs (graphs connecting nodes based on distance between them) to approximate nearest-neighbor search. To understand HNSW, we can break it down into 2 parts:

- **Hierarchical (H):** The algorithm operates over multiple layers
- **Navigable Small World (NSW):** Each vector is a node within a graph and is connected to several other nodes

### Hierarchical

The hierarchical aspect of HNSW builds off of the idea of skip lists.

Skip lists are multi-layer linked lists. The bottom layer is a regular linked list connecting an ordered sequence of elements. Each new layer above removes some elements from the underlying layer (based on a fixed probability), producing a sparser subsequence that “skips” over elements.

<div>
<img
alt="visual of an example skip list (light)"
className="dark:hidden"
src="/docs/img/ai/vector-indexes/hnsw-indexes/skip-list--light.png"
/>
<img
alt="visual of an example skip list (dark)"
className="hidden dark:block"
src="/docs/img/ai/vector-indexes/hnsw-indexes/skip-list--dark.png"
/>
</div>

When searching for an element, the algorithm begins at the top layer and traverses its linked list horizontally. If the target element is found, the algorithm stops and returns it. Otherwise if the next element in the list is greater than the target (or NIL), the algorithm drops down to the next layer below. Since each layer below is less sparse than the layer above (with the bottom layer connecting all elements), the target will eventually be found. Skip lists offer O(log n) average complexity for both search and insertion/deletion.

### Navigable Small World

A navigable small world (NSW) is a special type of proximity graph that also includes long-range connections between nodes. These long-range connections support the “small world” property of the graph, meaning almost every node can be reached from any other node within a few hops. Without these additional long-range connections, many hops would be required to reach a far-away node.

<img
alt="visual of an example navigable small world graph"
src="/docs/img/ai/vector-indexes/hnsw-indexes/nsw.png"
/>

The “navigable” part of NSW specifically refers to the ability to logarithmically scale the greedy search algorithm on the graph, an algorithm that attempts to make only the locally optimal choice at each hop. Without this property, the graph may still be considered a small world with short paths between far-away nodes, but the greedy algorithm tends to miss them. Greedy search is ideal for NSW because it is quick to navigate and has low computational costs.

### **Hierarchical +** Navigable Small World

HNSW combines these two concepts. From the hierarchical perspective, the bottom layer consists of a NSW made up of short links between nodes. Each layer above “skips” elements and creates longer links between nodes further away from each other.

Just like skip lists, search starts at the top layer and works its way down until it finds the target element. However instead of comparing a scalar value at each layer to determine whether or not to descend to the layer below, a multi-dimensional distance measure (such as Euclidean distance) is used.

## When should you create HNSW indexes?

Unlike IVFFlat indexes, you are safe to build an HNSW index immediately after the table is created. HNSW indexes are based on graphs which inherently are not affected by the same limitations as IVFFlat. As new data is added to the table, the index will be filled automatically and the index structure will remain optimal.

## Resources

Read more about indexing on `pgvector`'s [GitHub page](https://github.com/pgvector/pgvector#indexing).

export const Page = ({ children }) => <Layout meta={meta} children={children} />

export default Page
Original file line number Diff line number Diff line change
@@ -1,90 +1,99 @@
import Layout from '~/layouts/DefaultGuideLayout'

export const meta = {
id: 'ai-managing-indexes',
title: 'Managing indexes',
description: 'Understanding vector indexes',
sidebar_label: 'Managing indexes',
id: 'ai-ivf-indexes',
title: 'IVFFlat indexes',
description: 'Understanding IVFFlat indexes in pgvector',
sidebar_label: 'IVFFlat indexes',
}

Once your vector table starts to grow, you will likely want to add an index to speed up queries. Without indexes, you'll be performing a sequential scan which can be a resource-intensive operation when you have many records.
IVFFlat is a type of vector index for approximate nearest neighbor search, used in high-dimensional spaces like those found in embeddings.

## IVFFlat indexes
## Choosing an index

Today `pgvector` indexes use an algorithm called IVFFlat. IVF stands for 'inverted file indexes'. It works by clustering your vectors in order to reduce the similarity search scope. Rather than comparing a vector to every other vector, the vector is only compared against vectors within the same cell cluster (or nearby clusters, depending on your configuration).
Today `pgvector` supports two types of indexes:

### Inverted lists (cell clusters)
- [HNSW](/docs/guides/ai/vector-indexes/hnsw-indexes)
- [IVFFlat](/docs/guides/ai/vector-indexes/ivf-indexes)

When you create the index, you choose the number of inverted lists (cell clusters). Increase this number to speed up queries, but at the expense of recall.
In general we recommend using [HNSW](/docs/guides/ai/vector-indexes/hnsw-indexes) because of its [performance gain](/blog/increase-performance-pgvector-hnsw#hnsw-performance-1536-dimensions) and [robustness against changing data](/docs/guides/ai/vector-indexes/hnsw-indexes#when-should-you-create-hnsw-indexes). If you have a special use case that requires IVFFlat instead, keep reading.

For example, to create an index with 100 lists on a column that uses the cosine operator:
## Usage

```sql
create index on items using ivfflat (column_name vector_cosine_ops) with (lists = 100);
```
The way you create an IVFFlat index depends on the distance operator you are using. `pgvector` includes 3 distance operators:

For more info on the different operators, see [Distance operations](#distance-operators).
| Operator | Description | [**Operator class**](https://www.postgresql.org/docs/current/sql-createopclass.html) |
| -------- | ---------------------- | ------------------------------------------------------------------------------------ |
| `<->` | Euclidean distance | `vector_l2_ops` |
| `<#>` | negative inner product | `vector_ip_ops` |
| `<=>` | cosine distance | `vector_cosine_ops` |

For every query, you can set the number of probes (1 by default). The number of probes corresponds to the number of nearby cells to probe for a match. Increase this for better recall at the expense of speed.
Use the following SQL commands to create an IVFFlat index for the operator(s) used in your queries.

To set the number of probes for the duration of the session run:
### Euclidean L2 distance (`vector_l2_ops`)

```sql
set ivfflat.probes = 10;
create index on items using ivfflat (column_name vector_l2_ops) with (lists = 100);
```

To set the number of probes only for the current transaction run:
### Inner product (`vector_ip_ops`)

```sql
begin;
set local ivfflat.probes = 10;
select ...
commit;
create index on items using ivfflat (column_name vector_ip_ops) with (lists = 100);
```

If the number of probes is the same as the number of lists, exact nearest neighbor search will be performed and the planner won't use the index.
### Cosine distance (`vector_cosine_ops`)

### Approximate nearest neighbor
```sql
create index on items using ivfflat (column_name vector_cosine_ops) with (lists = 100);
```

One important note with IVF indexes is that nearest neighbor search is approximate, since exact search on high dimensional data can't be indexed efficiently. This means that similarity results will change (slightly) after you add an index (trading recall for speed).
Currently vectors with up to 2,000 dimensions can be indexed.

If you are using the `vecs` Python library, follow the instructions in [Managing collections](/docs/guides/ai/managing-collections#create-an-index) to create indexes.

## Distance operators
## How does IVFFlat work?

The type of index required depends on the distance operator you are using. `pgvector` includes 3 distance operators:
IVF stands for 'inverted file indexes'. It works by clustering your vectors in order to reduce the similarity search scope. Rather than comparing a vector to every other vector, the vector is only compared against vectors within the same cell cluster (or nearby clusters, depending on your configuration).

| Operator | Description | [**Operator class**](https://www.postgresql.org/docs/current/sql-createopclass.html) |
| -------- | ---------------------- | ------------------------------------------------------------------------------------ |
| `<->` | Euclidean distance | `vector_l2_ops` |
| `<#>` | negative inner product | `vector_ip_ops` |
| `<=>` | cosine distance | `vector_cosine_ops` |
### Inverted lists (cell clusters)

Use the following SQL commands to create an index for the operator(s) used in your queries.
When you create the index, you choose the number of inverted lists (cell clusters). Increase this number to speed up queries, but at the expense of recall.

### Euclidean L2 distance (`vector_l2_ops`)
For example, to create an index with 100 lists on a column that uses the cosine operator:

```sql
create index on items using ivfflat (column_name vector_l2_ops) with (lists = 100);
create index on items using ivfflat (column_name vector_cosine_ops) with (lists = 100);
```

### Inner product (`vector_ip_ops`)
For more info on the different operators, see [Distance operations](#distance-operators).

For every query, you can set the number of probes (1 by default). The number of probes corresponds to the number of nearby cells to probe for a match. Increase this for better recall at the expense of speed.

To set the number of probes for the duration of the session run:

```sql
create index on items using ivfflat (column_name vector_ip_ops) with (lists = 100);
set ivfflat.probes = 10;
```

### Cosine distance (`vector_cosine_ops`)
To set the number of probes only for the current transaction run:

```sql
create index on items using ivfflat (column_name vector_cosine_ops) with (lists = 100);
begin;
set local ivfflat.probes = 10;
select ...
commit;
```

Currently vectors with up to 2,000 dimensions can be indexed.
If the number of probes is the same as the number of lists, exact nearest neighbor search will be performed and the planner won't use the index.

If you are using the `vecs` Python library, follow the instructions in [Managing collections](/docs/guides/ai/managing-collections#create-an-index) to create indexes.
### Approximate nearest neighbor

One important note with IVF indexes is that nearest neighbor search is approximate, since exact search on high dimensional data can't be indexed efficiently. This means that similarity results will change (slightly) after you add an index (trading recall for speed).

## When should you add indexes?
## When should you create IVFFlat indexes?

`pgvector` recommends adding indexes only after the table has sufficient data, so that the internal IVFFlat cell clusters are based on your data's distribution. Anytime the distribution changes significantly, consider recreating indexes.
`pgvector` recommends buliding IVFFlat indexes only after the table has sufficient data, so that the internal IVFFlat cell clusters are based on your data's distribution. Anytime the distribution changes significantly, consider rebuilding indexes.

## Resources

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion apps/docs/public/sitemap.xml
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@
</url>

<url>
<loc>https://supabase.com/docs/guides/ai/managing-indexes</loc>
<loc>https://supabase.com/docs/guides/ai/vector-indexes</loc>
<changefreq>weekly</changefreq>
<changefreq>0.5</changefreq>
</url>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Without indexes, pgvector performs a full table scan when you run a similarity q

To solve this, pgvector offers indexes. Indexes reorganize the data into data structures that exploit internal structure and enable approximate similarity search without referring to every record. Currently, pgvector supports an IVF index, with HNSW expected in the next release.

IVF [indexes](https://supabase.com/docs/guides/ai/managing-indexes) work by clustering vectors into `lists`, and then querying only vectors within the same list (or multiple nearby lists, depending on the value of `probes`).
IVF [indexes](https://supabase.com/docs/guides/ai/vector-indexes/ivf-indexes#how-does-ivfflat-work) work by clustering vectors into `lists`, and then querying only vectors within the same list (or multiple nearby lists, depending on the value of `probes`).

### Scaling indexes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ USING hnsw (embedding vector_ip_ops);

## How does HNSW work?

Compared to inverted file (IVF) indexes which use [clusters](https://supabase.com/docs/guides/ai/managing-indexes#ivfflat-indexes) to approximate nearest-neighbor search, HNSW uses proximity graphs (graphs connecting nodes based on distance between them). To understand HNSW, we can break it down into 2 parts:
Compared to inverted file (IVF) indexes which use [clusters](https://supabase.com/docs/guides/ai/vector-indexes/ivf-indexes#how-does-ivfflat-work) to approximate nearest-neighbor search, HNSW uses proximity graphs (graphs connecting nodes based on distance between them). To understand HNSW, we can break it down into 2 parts:

- **Hierarchical (H):** The algorithm operates over multiple layers
- **Navigable Small World (NSW):** Each vector is a node within a graph and is connected to several other nodes
Expand Down
5 changes: 5 additions & 0 deletions apps/www/lib/redirects.js
Original file line number Diff line number Diff line change
Expand Up @@ -2293,4 +2293,9 @@ module.exports = [
source: '/blog/pgvector-v0-5-0-hnsw',
destination: '/blog/increase-performance-pgvector-hnsw',
},
{
permanent: true,
source: '/docs/guides/ai/managing-indexes',
destination: '/docs/guides/ai/vector-indexes',
},
]

0 comments on commit 7042eb0

Please sign in to comment.