Skip to content

[Tutorial] Compound indexing and query optimization #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -402,7 +402,8 @@
"pages": [
"tutorials/client/data/overview",
"tutorials/client/data/cascading-delete",
"tutorials/client/data/sequential-id-mapping"
"tutorials/client/data/sequential-id-mapping",
"/tutorials/client/data/compound-indexing-query-performance"
]
}
]
Expand Down
139 changes: 139 additions & 0 deletions tutorials/client/data/compound-indexing-query-performance.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
---
title: Best Practices for Compound Indexing and Query Optimization in PowerSync
description: In this tutorial we will show you the Best Practices for Compound Indexing and Query Optimization using the [PowerSync Web SDK](https://docs.powersync.com/client-sdk-references/javascript-web).
sidebarTitle: Compound Indexing & Query Performances
---

## Introduction

This tutorial outlines findings and recommendations based on extensive testing of compound indexes, query execution, and table performance using the [PowerSync Web SDK](https://docs.powersync.com/client-sdk-references/javascript-web).
These best practices are designed to assist developers in optimizing their schemas and queries when working with large datasets.

## Key Findings

### 1. Compound Index Behavior

- Compound indexes significantly improve query performance when filters align with the indexed column order.
- Queries that skip leading columns in a compound index result in full table scans, negating performance benefits.
- Performance is consistent with SQLite's behavior as PowerSync uses SQLite under the hood.

### 2. Performance Benchmarks

<Note>
The following results were obtained using a table with 1999 columns and 50k entries
</Note>


#### Queries with Leading Column Filters

```sql
EXPLAIN QUERY PLAN SELECT * FROM dummy_table WHERE col1 = 'val1';
```

- **Execution Time:** 256 ms
- **Query Plan:** SEARCH dummy_table USING COMPOUND INDEX

#### Queries with Multiple Indexed Columns

```sql
EXPLAIN QUERY PLAN SELECT * FROM dummy_table WHERE col1 = 'val1' AND col2 = 'val2';
```

- **Execution Time:** 206
- **Query Plan:** SEARCH dummy_table USING COMPOUND INDEX

#### Queries Skipping Leading Columns

```sql
EXPLAIN QUERY PLAN SELECT * FROM dummy_table WHERE col2 = 'val2' AND col3 = 'val3';
```

- **Execution Time:** 556 ms
- **Query Plan:** SCAN dummy_table
#### Performance Summary

| Query Type | Execution Time | Query Plan |
|-----------------------------------|----------------|-------------------------------------|
| Leading Column Filters | 253 ms | SEARCH dummy_table USING COMPOUND INDEX |
| Multiple Indexed Columns | 206 ms | SEARCH dummy_table USING COMPOUND INDEX |
| Skipping Leading Columns | 556 ms | SCAN dummy_table |
### 3. Column Limitations

- PowerSync supports a maximum of 1,999 columns in a compound index, aligning with SQLite's limitations.
- Queries including non-indexed columns slightly increase execution time but remain performant under typical workloads.

## Best Practices

**Faster query execution** can be achieved by following these best practices.

### Schema Definition

Define compound indexes to match your most frequent query patterns:

```javascript
import { column, Table } from '@powersync/web';

const todos = new Table(
{
list_id: column.text, // Foreign key to lists
created_at: column.text, // Creation timestamp
description: column.text, // Task description
},
{
indexes: {
list_created: ['list_id', 'created_at'],
},
}
);
```

### Query Optimization

#### Aligned Filters

Ensure that queries align with the column order of compound indexes:

```javascript
const results = await powerSync.get(
`SELECT * FROM todos WHERE list_id = ? AND created_at = ?`,
['list1', '2025-01-26']
);
```

#### Skipping Leading Columns

For queries skipping leading columns, define additional indexes:

```javascript
await powerSync.execute(
`CREATE INDEX idx_description ON todos (description);`
);
```

### Advanced Scenarios

#### Testing Column Order

The order of indexed columns affects query performance:

```javascript
// Create compound index with a different order
await powerSync.execute(`CREATE INDEX idx_order ON todos (description, list_id);`);

// Query execution profiling
console.time('Query with reordered index');
await powerSync.get(
`SELECT * FROM todos WHERE description = ? AND list_id = ?`,
['Task 1', 'list1']
);
console.timeEnd('Query with reordered index');
```

## Key Takeaways

- Define compound indexes aligned with your most frequent query patterns.
- Avoid skipping leading columns in compound indexes; create additional indexes if necessary.

## Conclusion

By adhering to these best practices and utilizing PowerSync’s schema capabilities, applications can achieve significant performance improvements whilst ensuring scalability.
1 change: 1 addition & 0 deletions tutorials/client/data/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@ description: "A collection of tutorials showcasing various data management strat
<CardGroup>
<Card title="Cascading Delete" icon="database" href="/tutorials/client/data/cascading-delete" horizontal/>
<Card title="Sequential ID Mapping" icon="database" href="/tutorials/client/data/sequential-id-mapping" horizontal/>
<Card title="Compound Indexing & Query Performance" icon="database" href="/tutorials/client/data/compound-indexing-query-performance" horizontal/>
</CardGroup>