-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Audit serde of array metadata #396
Comments
I'm gonna try making the Validity metadata for Structs much smaller. |
We might eventually want to squeeze all metadata into 32-bits. We can reserve 0xffffffff to indicate that the metadata has spilled into a buffer. |
I think we can spare 64 bits per encoding |
For most arrays, validity metadata is just a single bit for whether or not a validity child is defined. |
RunEndremove length, dtype => ptype (it has to be an int).
ALP
RunEndBoolremove length, dtype => ptype.
RoaringInt
FoRScalar => ScalarValue, use self.dtype(). Buffer, BufferString, List should go into the Array buffer.
DictDType => PType
DateTimePartsDType => PType.
FSSTDType => PType.
Nullremove len.
Primitive
VarBinDType => PType
Delta
RoaringBoolRemove length
BitPackedRemove length.
ByteBool
ZigZag
ExtensionDType => PType
StructRemove length.
Chunked
Sparseremove len, DType => PType, Scalar => ScalarValue.
ConstantScalar => ScalarValue, remove length.
BoolRemove length.
VarBinView
|
We currently implement naive serde using Rust serde + flexbuffers by default.
Many arrays can pack their metadata much more tightly.
This is an overview issue to track auditing each one:
The text was updated successfully, but these errors were encountered: