Describe proposed blob structure #64
Conversation
|
|
||
| - the version (set to zero) (1 byte) | ||
| - header length in chunks, excluding this one (1 byte) | ||
| - I expect this to be zero most of the time |
There was a problem hiding this comment.
Why header length is zero most of the time?
| - offset to corresponding segment after accounting for the multiplier (2 bytes) | ||
| - Pad the remaining space with zeroes. Note that zero is an invalid data identifier, so there should be no confusion about what counts as padding. | ||
|
|
||
| If the header is larger than one chunk, retrieve and validate the remaining chunks. They should all be structured with the same data type records (up to 6 in a chunk) and padded with zeros. |
There was a problem hiding this comment.
if there are more than one blobs in a publication, will only the first blob contains the header (if the header is small enough to fit into a blob)?
|
|
||
| Although we are not using binary search, I still think it is useful to have a deterministic structure. Therefore, in addition to validating the header structure, the rollup nodes should ensure: | ||
|
|
||
| - all data identifiers are listed in ascending order |
There was a problem hiding this comment.
If we do not use binary search, will the order of data identifiers matter?
|
|
||
| The data identifier will indicate which kind of compression (eg. gzip, lz4, etc) was used. After decompression, the segment should use the same structure so that it can be further subdivided. For simplicity: | ||
|
|
||
| - the segment header should be outside the compression. This means the expected contents can be identified (and potentially ignored) before retrieving the whole segment and decompressing it. |
There was a problem hiding this comment.
there is no definition of "segment header"...
| - the segment header should be outside the compression. This means the expected contents can be identified (and potentially ignored) before retrieving the whole segment and decompressing it. | ||
| - we should disallow nested compressed segments. I suspect they are unlikely to achieve much (you can always define the data identifier as multiple rounds of a compression algorithm) and we don't want the node to have to recurse indefinitely only to find the publication is poorly formed. | ||
|
|
||
| ### Rollup segments |
There was a problem hiding this comment.
If a rollup consists of multiple segments, should it concatenate these segments (excluding their headers) into a single byte array and validate them as a whole, or should each segment be validated individually, with the results (potentially structured) then concatenated together?
| Although we are not using binary search, I still think it is useful to have a deterministic structure. Therefore, in addition to validating the header structure, the rollup nodes should ensure: | ||
|
|
||
| - all data identifiers are listed in ascending order | ||
| - the actual data segments are also in the same order (i.e. the segment offsets are strictly increasing) |
There was a problem hiding this comment.
I prefer not to use the next segment’s offset to determine the current segment’s size. This way, app A can use segment [100, 200], while app B can use segment [100, 300], allowing for greater flexibility.
No description provided.