Refactor Ref System with Nested Optional Properties #386

samwillis · 2025-08-07T15:16:17Z

🎯 Overview

This PR introduces a comprehensive refactor of the ref system to properly support nested structures and optionality, bringing the type system in line with JavaScript's optional chaining behavior.

✨ Key Improvements

🔗 Nested Optional Properties: Full support for deeply nested optional objects (employees.profile?.bio, orders.customer?.address?.street)
🛡️ Enhanced Type Safety: Optional types now correctly typed as RefProxy<T> | undefined with optionality outside the ref, enabling proper optional chaining
🔧 New Query Functions: Added isUndefined, isNotUndefined, isNull, isNotNull for proper null/undefined checks in queries
⚙️ Improved JOIN Handling: Fixed optionality in JOIN operations and multiple GROUP BY clause support

🚨 Breaking Changes

IMPORTANT: Code that previously worked by ignoring optionality now requires proper optional chaining syntax.

// Before (worked but was type-unsafe)
employees.profile.bio         // ❌ Now throws type error

// After (correct and type-safe)  
employees.profile?.bio        // ✅ Required syntax

🔧 Technical Changes

Type System Refactor

RefProxyForContext: Now handles optionality outside refs (RefProxy<T> | undefined)
PrecomputeRefStructure: Only wraps leaf values, leaving objects as plain TypeScript
Enhanced SelectValue: Support for nested projections and JavaScript literals
New Helper Types: Comprehensive null/undefined handling

Runtime Improvements

Compiler Functions: Implemented isNotUndefined, isUndefined, isNull, isNotNull
GROUP BY Fixes: Multiple .groupBy() calls now accumulate instead of replace
JOIN Optionality: Correct handling of optional tables in different join types

📊 Testing

✅ 1,050 tests passing (60 test files)
✅ 0 type errors
✅ 0 linting errors
✅ Enhanced type safety with comprehensive nested property coverage

🗂️ Files Changed

Core Type System

packages/db/src/query/builder/types.ts - Major refactor of ref types
packages/db/src/query/builder/index.ts - Enhanced query builder methods
packages/db/src/query/builder/functions.ts - New null/undefined functions

Compiler & Runtime

packages/db/src/query/compiler/evaluators.ts - Added missing function implementations
packages/db/src/query/compiler/group-by.ts - Fixed multiple GROUP BY validation

Tests & Documentation

Updated 100+ test files to use proper optional chaining syntax
Fixed linting issues from improved type inference
Added comprehensive changeset documentation

🚀 Migration Guide

For existing codebases: Add ?. when accessing potentially undefined nested properties. The TypeScript compiler will guide you to the correct syntax.

🎉 Benefits

Eliminates runtime errors from accessing undefined nested properties
Better IDE experience with cleaner type displays and proper IntelliSense
JavaScript alignment - type system now mirrors optional chaining behavior
Improved SQL compliance with better validation and null handling

This enhancement significantly improves developer experience while maintaining the robustness and performance of the existing query system.

Added comprehensive test coverage for querying nested objects across WHERE, SELECT, GROUP BY, and ORDER BY operations. Fixed query builder syntax issues by removing JavaScript expressions, null coalescing operators, and unnecessary optional chaining. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

changeset-bot · 2025-08-07T15:16:21Z

🦋 Changeset detected

Latest commit: 8eae2a8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages

Name	Type
@tanstack/db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

pkg-pr-new · 2025-08-07T15:17:41Z

More templates

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@386

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@386

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@386

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@386

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@386

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@386

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@386

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@386

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@386

commit: 8eae2a8

github-actions · 2025-08-07T15:18:49Z

Size Change: +99 B (+0.15%)

Total Size: 64.5 kB

Filename	Size	Change
`./packages/db/dist/esm/query/builder/functions.js`	584 B	+9 B (+1.57%)
`./packages/db/dist/esm/query/builder/index.js`	3.82 kB	+27 B (+0.71%)
`./packages/db/dist/esm/query/compiler/evaluators.js`	1.54 kB	+63 B (+4.25%)

ℹ️ View Unchanged

Filename	Size
`./packages/db/dist/esm/change-events.js`	1.13 kB
`./packages/db/dist/esm/collection.js`	10.6 kB
`./packages/db/dist/esm/deferred.js`	230 B
`./packages/db/dist/esm/errors.js`	3 kB
`./packages/db/dist/esm/index.js`	1.52 kB
`./packages/db/dist/esm/indexes/auto-index.js`	745 B
`./packages/db/dist/esm/indexes/base-index.js`	605 B
`./packages/db/dist/esm/indexes/btree-index.js`	1.74 kB
`./packages/db/dist/esm/indexes/lazy-index.js`	1.25 kB
`./packages/db/dist/esm/local-only.js`	827 B
`./packages/db/dist/esm/local-storage.js`	2.03 kB
`./packages/db/dist/esm/optimistic-action.js`	294 B
`./packages/db/dist/esm/proxy.js`	4.19 kB
`./packages/db/dist/esm/query/builder/ref-proxy.js`	890 B
`./packages/db/dist/esm/query/compiler/expressions.js`	631 B
`./packages/db/dist/esm/query/compiler/group-by.js`	2.06 kB
`./packages/db/dist/esm/query/compiler/index.js`	2.15 kB
`./packages/db/dist/esm/query/compiler/joins.js`	2.36 kB
`./packages/db/dist/esm/query/compiler/order-by.js`	1.17 kB
`./packages/db/dist/esm/query/compiler/select.js`	655 B
`./packages/db/dist/esm/query/ir.js`	466 B
`./packages/db/dist/esm/query/live-query-collection.js`	333 B
`./packages/db/dist/esm/query/live/collection-config-builder.js`	2.51 kB
`./packages/db/dist/esm/query/live/collection-subscriber.js`	2.32 kB
`./packages/db/dist/esm/query/optimizer.js`	2.63 kB
`./packages/db/dist/esm/SortedMap.js`	1.24 kB
`./packages/db/dist/esm/transactions.js`	2.29 kB
`./packages/db/dist/esm/utils.js`	419 B
`./packages/db/dist/esm/utils/btree.js`	6.02 kB
`./packages/db/dist/esm/utils/comparison.js`	718 B
`./packages/db/dist/esm/utils/index-optimization.js`	1.62 kB

_{compressed-size-action::db-package-size}

github-actions · 2025-08-07T15:19:57Z

Size Change: 0 B

Total Size: 1.16 kB

ℹ️ View Unchanged

Filename	Size
`./packages/react-db/dist/esm/index.js`	152 B
`./packages/react-db/dist/esm/useLiveQuery.js`	1.01 kB

_{compressed-size-action::react-db-package-size}

kevin-dp · 2025-08-19T08:06:55Z

@samwillis Do we really need isNotUndefined and isNotNull? I would expect to write that as not(isUndefined(...)) and not(isNull(...)).

samwillis · 2025-08-19T08:09:24Z

@samwillis Do we really need isNotUndefined and isNotNull? I would expect to write that as not(isUndefined(...)) and not(isNull(...)).

Agreed, lets drop them

kevin-dp

I did a first pass over the core changes. Will review the changes to the tests after lunch.

packages/db/src/query/builder/types.ts

packages/db/src/query/builder/functions.ts

packages/db/src/query/builder/types.ts

kevin-dp

The tests look good to me. Let's fix my comments where possible before we merge.

packages/db/tests/query/basic.test.ts

KyleAMathews and others added 16 commits August 7, 2025 11:02

checkpoint

f44031d

checkpoint

57ec777

checkpoint

9f57dd6

checkpoint

e96bafd

checkpoint

3b26333

checkpoint

efa4b13

checkpoint

9f12edb

checkpoint

d5921a4

checkpoint

6c56f1d

checkpoint

046ad7d

checkpoint

9ca2797

checkpoint

818cd7b

checkpoint

7a077a8

checkpoint

ae11e1d

tests and lint pass!!

9f8b947

samwillis requested a review from kevin-dp August 7, 2025 15:16

changeset

c3a305e

samwillis mentioned this pull request Aug 7, 2025

Add nested object query tests #352

Closed

samwillis force-pushed the ref-refactor branch from 1b1345b to c3a305e Compare August 7, 2025 15:19

samwillis mentioned this pull request Aug 9, 2025

Types tracking #385

Open

samwillis added 3 commits August 10, 2025 16:30

improve ref types in ide

cfb0b9e

tidy

73ea1ee

tidy functions

110f9d6

kevin-dp reviewed Aug 19, 2025

View reviewed changes

kevin-dp requested changes Aug 19, 2025

View reviewed changes

packages/db/tests/query/basic.test.ts Outdated Show resolved Hide resolved

kevin-dp mentioned this pull request Aug 19, 2025

Refactor select improving spread (...obj) and enabling nested projection. #389

Open

samwillis added 5 commits August 26, 2025 15:21

wip review fixes

bec6ea5

finish changes from review

ec34933

test

ac8809b

Merge branch 'main' into ref-refactor

3643e9c

fix tests after merging in main - they were missing optionality handling

8eae2a8

samwillis requested a review from kevin-dp August 26, 2025 16:22

kevin-dp approved these changes Aug 27, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor Ref System with Nested Optional Properties #386

Refactor Ref System with Nested Optional Properties #386

Uh oh!

samwillis commented Aug 7, 2025

Uh oh!

changeset-bot bot commented Aug 7, 2025 •

edited

Loading

Uh oh!

pkg-pr-new bot commented Aug 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 7, 2025 •

edited

Loading

Uh oh!

kevin-dp commented Aug 19, 2025

Uh oh!

samwillis commented Aug 19, 2025

Uh oh!

kevin-dp left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevin-dp left a comment

Uh oh!

Uh oh!

Uh oh!

Refactor Ref System with Nested Optional Properties #386

Are you sure you want to change the base?

Refactor Ref System with Nested Optional Properties #386

Uh oh!

Conversation

samwillis commented Aug 7, 2025

🎯 Overview

✨ Key Improvements

🚨 Breaking Changes

🔧 Technical Changes

Type System Refactor

Runtime Improvements

📊 Testing

🗂️ Files Changed

Core Type System

Compiler & Runtime

Tests & Documentation

🚀 Migration Guide

🎉 Benefits

Uh oh!

changeset-bot bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

pkg-pr-new bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevin-dp commented Aug 19, 2025

Uh oh!

samwillis commented Aug 19, 2025

Uh oh!

kevin-dp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevin-dp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

changeset-bot bot commented Aug 7, 2025 •

edited

Loading

pkg-pr-new bot commented Aug 7, 2025 •

edited

Loading

github-actions bot commented Aug 7, 2025 •

edited

Loading

github-actions bot commented Aug 7, 2025 •

edited

Loading