Web UI stops responding when ZFS snapshot are created manually

[dsolsona]

Hi folks,

I'm experimenting with the community version and today I found an issue while I was playing around with ZFS.

I don't really know what I am doing 😆 so it is quite possible we are not meant to create snapshots manually.

I have the following filesystem structure

root@postgres-anon:~# zfs list
NAME                                                 USED  AVAIL  REFER  MOUNTPOINT
dblab_pool                                          64.2G   225G  64.1G  /var/lib/dblab/dblab_pool
dblab_pool/branch                                   1.20M   225G    96K  /var/lib/dblab/dblab_pool/branch
dblab_pool/branch/main                              1.11M   225G   104K  /var/lib/dblab/dblab_pool/branch/main
dblab_pool/branch/main/sandbox-v2-long-duration     1.01M   225G    96K  /var/lib/dblab/dblab_pool/branch/main/sandbox-v2-long-duration
dblab_pool/branch/main/sandbox-v2-long-duration/r0   936K   225G   334M  /var/lib/dblab/dblab_pool/branch/main/sandbox-v2-long-duration/r0

I was manually running a postgres instance with /var/lib/dblab/dblab_pool mounted in and I manually restored a backup I had to test some things. Then I was trying to figure out how could I get a new snapshot so any future clones gets the latest data I restored, instead of the initial data.

So I went and created a new snapshot.

root@postgres-anon:~# zfs list -t snapshot
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
dblab_pool@snapshot_20250721202933   131M      -   334M  -
dblab_pool@snapshot_20250723091600  1.91M      -  64.1G  -

dblab_pool@snapshot_20250721202933 is the first snapshot that was created when I ran the dblab server.

dblab_pool@snapshot_20250723091600 is the one I created manually.

After restarting the dblab-server container when I tried to access the console I got a blank screen.

I saw the following javascript error in the browser console

Snapshots.ts:87 RangeError: Invalid time value
    at index.tsx:141:24
    at Array.map (<anonymous>)
    at index.tsx:121:34

index.js:174 Uncaught (in promise) RangeError: Invalid time value
    at index.tsx:141:24
    at Array.map (<anonymous>)
    at index.tsx:121:34

Deleting the manual snapshot and restarting the container works and I get access back.

A few questions:

1. How can I create a new snapshot for dblab_pool ? The UI only allows me to create snapshots from clones, but I don't want that
2. Alternative, how do I refresh the dblab_pool@snapshot_20250721202933 snapshot? I can't delete it
3. I can't crete a new branch, because it would be based on the existing snapshot which I don't want.

So basically, I'll have a process that every few hours updates /var/lib/dblab/dblab_pool/data with the latest data and once that happens, I want to have a new snapshot available so I can created new clonse from it. In the near future I'll most likely have a running postgres instance that replicates data from production and anonymises it, so I will want to be able to generate frequent snapshots.

Thanks for the awesome product and your future help!

[agneum]

Hi @dsolsona Thanks for the report, we will check this case.

By creating manually, you mean using a create snapshot endpoint or sub-command via API/CLI, and not just the zfs snapshot .. command directly in the terminal, right?

1 and 2. Snapshots are read-only entities, they cannot be updated. You can only create a newer snapshot.
As far as I understand, you are using the logical data recovery mode.

The standard mechanism looks like deleting the current data from the pool (snapshots and clones) and restoring a more recent version and creating a snapshot of this version.

In this case, you can configure a full refresh, as described in the tutorial:
https://postgres.ai/docs/dblab-howtos/administration/logical-full-refresh#automatic-full-refresh-data-from-a-source

Moreover, you do not even need to have several pools, you can create several datasets on one pool using the zfs create .. command.

3. A branch is a pointer to a snapshot. A snapshot of a new branch is always based on the previous snapshot. Branches without snapshots are impossible.

Perhaps, the physical recovery mode is more suitable for your instance. There, snapshots are performed on a schedule by default:
https://postgres.ai/docs/dblab-howtos/#physical-retrieval
Use the most convenient tool for you to restore

Or some hybrid option: you are responsible for data delivery (dump/restoration), and the engine only takes snapshots. In this case, you can specify only those jobs that you need, that is, only logicalSnapshot, omitting dump and restore jobs:
https://postgres.ai/docs/reference-guides/database-lab-engine-configuration-reference#section-retrieval-data-retrieval

Thank you very much for the feedback.

[dsolsona]

Hi Artyom,

Thanks for your update 🙇

By creating manually, you mean using a create snapshot endpoint or sub-command via API/CLI, and not just the zfs snapshot .. command directly in the terminal, right?

I created the snapshot using the ZFS cli.

1 and 2. Snapshots are read-only entities, they cannot be updated. You can only create a newer snapshot.
As far as I understand, you are using the logical data recovery mode.

right, that makes sense.

The standard mechanism looks like deleting the current data from the pool (snapshots and clones) and restoring a more recent version and creating a snapshot of this version.

that would not be an option because it means deleting all the clones. For my particular use case, the clones can be long lived, they can be there for weeks or months. We create ephemeral environments with anonymised production data that are used by customers to test certain things. These ephemeral environments sometimes need to be updated to fetch the latest anonymised data.

So when I update the data in the base branch, I need to keep my clones, and have a process to update the data for them.

A branch is a pointer to a snapshot. A snapshot of a new branch is always based on the previous snapshot. Branches without snapshots are impossible.

Right. In the UI creating branches is based on a snapshot. So I'm back to my initial problem, which is getting a new snapshot in place. I don't see an option in the UI for creating a snapshot from the /var/lib/dblab/dblab_pool dataset.

And if I create one manually I get the crash in the UI 😅

I'll take a look at the links you shared with me, but I'm a bit confused with the different modes of operation. My mental model and what I would love to achieve is:

1. Update the base dataset /var/lib/dblab/dblab_pool with the latest anonymised data
2. Take a new snapshot
3. Selectively update clones with the data from the latest snapshot I took.
4. Keep other clones using the old data

If I do this via a new branch or an existing one would not be a problem. I guess one option could be to create a new branch from the new snapshot, delete the old clone, create the clone again from the new branch.

I'm still trying to build the correct mental model, as I can see many different ways to get this done.

[agneum]

Can you please clarify how exactly you manually restored the data?

Got it, thanks.

The fact is that the DBLab recovery process adds custom metadata to snapshots, so that it has correct timestamps. If they were lost or corrupted, a manually created snapshot may contain an incomplete or incorrect set of properties, which leads to errors in UI

I would recommend using DBLab CLI or API calls to accurately fill in all the snapshot metadata

[dsolsona]

Can you please clarify how exactly you manually restored the data?

Got it, thanks.

The fact is that the DBLab recovery process adds custom metadata to snapshots, so that it has correct timestamps. If they were lost or corrupted, a manually created snapshot may contain an incomplete or incorrect set of properties, which leads to errors in UI

I would recommend using DBLab CLI or API calls to accurately fill in all the snapshot metadata

Yeah I realized that there's a property added to the snapshots that seems to be used in the UI. That would explain the issue I saw.

It might be a good idea to add some safety checks to ignore snapshots if they do not contain the property, at least to avoid the UI crashing if anyone else decides to do a manual snapshot for whatever reason.