Document MaybeUninit bit validity

[joshlf]

Partially addresses rust-lang/unsafe-code-guidelines#555 by clarifying that it is sound to write any byte values (initialized or uninitialized) to any MaybeUninit<T> regardless of T.

r? @RalfJung

[joshlf]

Moving this discussion here:

The MaybeUninit docs probably make sense for this. We now do have a definition of "byte" in the reference that this can link to.

Okay, awesome. And what wording would you recommend? Would it be accurate to say something like the following?

The value of a [MaybeUninit<u8>; N] may contain pointer provenance, and so p: P -> [MaybeUninit<u8>; N] -> P preserves the value of p, including provenance

@RalfJung would you like me to add language like this to this PR?

[joshlf]

Update: I've added the following as a more concrete and fleshed out draft. I can edit or remove as preferred.

/// # Provenance
///
/// `MaybeUninit` values may contain [pointer provenance][provenance]. Concretely, for any
/// pointer type, `P`, which contains provenance, transmuting `p: P` to
/// `MaybeUninit<[u8; size_of::<P>]>` and then back to `P` will produce a value identical to
/// `p`, including provenance.
///
/// [provenance]: ../ptr/index.html#provenance

[joshlf]

Updated to reference the definition of a byte.

[RalfJung]

Cc @rust-lang/opsem

[RalfJung]

I don't think it makes sense to say that a type "contains initialized bytes" at some offset. That's a property of a representation.

The typical term for representation bytes that are lost here is "padding". I don't think we have rigorously defined padding anywhere yet, but the term is sufficiently widely-used (and generally with a consistent meaning) that we may just be able to use it here?

[joshlf]

IIUC, you're making two points:

• We should speak about a type's representation containing bytes, not about the type itself containing bytes
• In a representation, we should speak about padding bytes rather than uninitialized bytes

Is that right?

One thing that's probably worth distinguishing here is between values and layouts. In my mental model, an uninit byte is one of the possible values that a byte can have (e.g., it's the 257th value that can legally appear in a MaybeUninit<u8>). By contrast, padding is a property of a layout - namely, it's a sequence of bytes in a type's layout that happen to have the validity [MaybeUninit<u8>; PADDING_LEN].

Based on this, maybe it's best to say:

If byte offsets exists at which T's representation does not permit uninitialized bytes but U's representation does (e.g. due to padding), then the bytes in T at these offsets may not be preserved in u, and so transmute(u) may produce a T with uninitialized bytes at these offsets. This is an active area of discussion, and this code may become sound in the future.

[RalfJung]

Is that right?

No. I think both of the following concepts make sense:

• The representation of a particular value at a particular type contains uninitialized bytes.
• A type contains padding bytes. (These are bytes which are always ignored by the representation relation.)

But it makes less sense to talk about padding of a representation, or to talk about uninitialized bytes in a type.

So for this PR, the two key points (and they are separate points) are:

• If U has padding, those bytes may be reset to "uninitialized" as part of the round-trip. If those same bytes are not padding in T, this can therefore mean some of the information of the original T value is lost.
• If T does not permit uninitialized bytes on those positions, the round-trip is UB.

The second point is just a logical consequence of the first, it does not add any new information. Not sure if it is worth mentioning.

[joshlf]

• The representation of a particular value at a particular type contains uninitialized bytes.
• A type contains padding bytes. (These are bytes which are always ignored by the representation relation.)

Does this imply that a type contains padding bytes, not a type's representation?

I'm thinking through the implications of what you said, and I think I understand something new that I didn't before, and I want to run it by you: In my existing mental model, a padding byte is a location in a type's layout such that every byte value at that location (including uninit) is valid (enums complicate this model, but I don't think that complication is relevant for this discussion - we can just stick to thinking about structs). The problem with this mental model is that, interpreted naively, it implies that different byte values in a padding byte could correspond to different logical values of the type. So e.g. in the type #[repr(C)] struct T(u8, u16), [0, 0, 0, 0] and [0, 1, 0, 0] would correspond to different values of the type since we're treating the padding byte itself as part of the representation relation. Of course, that is not something we want.

IIUC, by contrast your model is that the representation relation simply doesn't include padding bytes at all. So it'd be more accurate to describe the representation of T as consisting of three bytes - at offsets 0, 2, and 3. Every representation of T has a "hole" at offset 1 which is not part of the representation. This ensures that there's a 1:1 mapping between logical values and representations. Is that right?

[RalfJung]

Does this imply that a type contains padding bytes, not a type's representation?

That's how I think about it. We can't tell which byte is a padding byte by looking at one representation -- it's a property of the type.

In my existing mental model, a padding byte is a location in a type's layout such that every byte value at that location (including uninit) is valid

That would make the only byte of MaybeUninit<u8> a padding byte, so I don't think this is the right definition.
That's why I said above: a padding byte is a byte that is ignored by the representation relation. Slightly more formally: if r is some representation valid for type T, and r' is equal to r everywhere except for padding bytes, then r and r' represent the same value.

So it'd be more accurate to describe the representation of T as consisting of three bytes

The representation has 4 bytes. But only 3 of them actually affect the represented value (which is a tuple of two [mathematical] integers).

---

We seem to be using the term "representation" slightly differently. For me, that's list a List<Byte> of appropriate length. You may be using that term to refer to what I call "representation relation"?

[joshlf]

We seem to be using the term "representation" slightly differently. For me, that's list a List<Byte> of appropriate length. You may be using that term to refer to what I call "representation relation"?

That's helpful, thank you!

To avoid rabbit holing too much on the definitions (although it's interesting and useful – just maybe a bit of a distraction here), maybe you could propose language you'd prefer to see in place of what I've written here?

[RalfJung]

@rustbot ready

[RalfJung]

@rustbot author

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[RalfJung]

I raised the question on Zulip whether it is wise to make a guarantee here that isn't, strictly speaking, documented in the LLVM LangRef. Nikita says he thinks that that's fine -- we may have to adjust how exactly we compile MaybeUninit in the future, but LLVM currently intends do support this case in a somewhat roundabout and incomplete way that seems to work well enough in practice, and LLVM can't more aggressively exploit the fuzziness along the edges of that approximation until a proper alternative exists.

[karolzwolak]

Thanks for your contribution @joshlf from wg-triage.
Could you address the comments above?

[joshlf]

Thanks for your contribution @joshlf from wg-triage. Could you address the comments above?

I likely won't have time to move this forward until mid-September or October, but I'll follow up at that point.

[joshlf]

Thanks for your contribution @joshlf from wg-triage. Could you address the comments above?

I likely won't have time to move this forward until mid-September or October, but I'll follow up at that point.

Nvm, found some time 🙂

I've responded to various comment threads.

[karolzwolak]

Awesome, you should also rebase your changes onto master, and do @rustbot ready when you're ready for review.

[joshlf]

@rustbot ready

[RalfJung]

Suggested change

	/// Note that, so long as no such byte offsets exist, then the preceding `identity` example *is* sound.
	/// However, so long as no such byte offsets exist, then the preceding `identity` example *is* sound.
	/// In particular, since `[u8; N]` has no padding bytes, transmuting `t` to `MaybeUninit<[u8; size_of::<T>]>`
	/// and back will always produce the original value `t` again. This is true even if `t` contains [provenance]:
	/// the resulting value will have the same provenance as the original `t`.
	///
	/// [provenance]: crate::ptr#provenance

And then we don't need the dedicated provenance section at all, do we?

[joshlf]

And then we don't need the dedicated provenance section at all, do we?

Done, and removed the provenance section.

[RalfJung]

Thanks, I like it! However, more eyes can't hurt -- @rust-lang/opsem @traviscross (since this is basically Reference-like material).

I also wonder what process we need to invoke, if any. This does document a new guarantee, after all. Nominating for t-lang.

[RalfJung]

There's a bit of a footgun for potential misunderstandings here but maybe I am being too nitpicky -- and I don't know what else we could say, anyway:

if T is &mut _ or &_, the resulting value will actually not be identical. The transmute itself is a perfect identity, but when a function return a reference (and this includes transmute returning a reference), that can influence the aliasing model. Tree Borrows will consider the resulting reference a child of the original reference.

[joshlf]

Maybe the core issue here is how we use the term "value"? E.g. I might say something like: "let y = x produces a y with the same value as x", but it's semantically relevant that there are now two copies of this value instead of one (e.g. image that x: &mut T). So maybe I'm using "value" in a way that doesn't capture everything about the value? More precisely, maybe I'm using "value" in a way that captures everything local about the value, but doesn't capture everything about the relationship between that value and other values (which is relevant when it comes to aliasing). Of course, provenance complicates this story because we talk about provenance "living inside" a particular value, but it's also a non-local property.

That doesn't really resolve your concern, but some random thoughts. Maybe it'll prompt you to think of better language we could use here?

[joshlf]

Another possibility: Instead of saying that the value is fully preserved, maybe we could say that the following contents of the value are preserved?

• Bit pattern
• Provenance

...and then explicitly disclaim any other value contents that we add to the AM in the future?

[RalfJung]

It's more a problem of abstraction: the Rust code let y = x; becomes something like y = x; Retag(y) in MIR, and that Retag does change the value. Specifically it changes the provenance, making y a child of x rather than wholly identical to x.

(In fact we sometimes even insert reborrows, making this more like y = &mut *x. I don't know the exact conditions for that.)

So maybe I'm using "value" in a way that doesn't capture everything about the value? More precisely, maybe I'm using "value" in a way that captures everything local about the value, but doesn't capture everything about the relationship between that value and other values (which is relevant when it comes to aliasing). Of course, provenance complicates this story because we talk about provenance "living inside" a particular value, but it's also a non-local property.

Alias tracking is based on provenance. Provenance is also just data. It may reference other data, such as indicating a position in a tree.

[joshlf]

Another possibility: Instead of saying that the value is fully preserved, maybe we could say that the following contents of the value are preserved?

• Bit pattern
• Provenance

...and then explicitly disclaim any other value contents that we add to the AM in the future?

In that case, I'm leaning towards this option unless you have thoughts about language we could use that captures specifically the subset of "value" that we want to address here. How does that sound?

[RalfJung]

Here's the best I can come up with so far. But I feel like this may actually be easier to write for someone who's less deeply entrenched in these discussions. ;)

Note: if t contains a reference, then there may be implicit reborrows of the reference any time it is copied, which may alter its provenance. In that case, the value returned by identity may not be exactly the same as its argument. However, even in this case, it remains true that identity behaves the same as a function that just returns t immediately.

[joshlf]

Okay here's my version of this (which is in the PR now):

Note a potential footgun: if t contains a reference, then there may be implicit reborrows of the reference any time it is copied, which may alter its provenance. In that case, while the value returned by identity is exactly the same as its argument, that value may immediately be reborrowed upon return, altering its provenance. This may make this call to identity behave as though it does not exactly preserve provenance.

[RalfJung]

while the value returned by identity is exactly the same as its argument

That part isn't given (there can be reborrows inside identity).

In fact my comment explicitly stated "the value returned by identity may not be exactly the same as its argument" so not sure how you got from there to your version.

It also seems very useful to state that it is still equivalent to a "boring" identity function.

[joshlf]

In fact my comment explicitly stated "the value returned by identity may not be exactly the same as its argument" so not sure how you got from there to your version.

I thought you were being overly-conservative in your wording, which I realize now isn't the case.

Changed to be closer to your wording. Better?

Note a potential footgun: if t contains a reference, then there may be implicit reborrows of the reference any time it is copied, which may alter its provenance. In that case, the value returned by identity may not be exactly the same as its argument. However, even in this case, it remains true that identity behaves the same as a function that just returns t immediately (i.e., fn identity<T>(t: T) -> T { t }).

[RalfJung]

That works for me :)

[traviscross]

Suggested change

	/// may not be preserved in `MaybeUninit<U>`. Interpreting the representation of `u` at type `T` again (i.e., `transmute(u)` above) may thus
	/// may not be preserved in `MaybeUninit<U>`. Interpreting the representation of `u` as type `T` again (i.e., `transmute(u)` above) may thus

[joshlf]

IIUC "at" is intentional language that @RalfJung uses – it's nomenclature used by language theorists.

[RalfJung]

"interpret u as type T" sounds wrong. We don't interpret u as a type, it's not a type. We use the type to tell us how to interpret u.

But I am not sure if there is widely-used standard terminology here, "interpret x at type y" just sounded most natural to me.

[traviscross]

Right. Reading more closely, to use "as" here I'd instead use "as of", e.g. "interpret the bytes of x as being of the type T". I.e., a value "has a type", "is of a type", or is treated or interpreted as "having" or "being of" a type.

I'm pretty happy to use PLT jargon in general, but for that to work we need to use the piece of jargon pervasively enough that people pick up on it (and probably discuss the jargon in our notation guide or elsewhere). If we just use it in one place, people will just think it's a typo.

[RalfJung]

I don't even know if this is PLT jargon or just Ralf jargon. ;)

[joshlf]

What if we instead say this?

Transmuting u to T (i.e., transmute(u) above) may thus...

[RalfJung]

That also works for me. I'd rather say "Transmuting u back to T" (to preserve the "again" in the current version).

[joshlf]

Done!

[traviscross]

We briefly reviewed this is the last lang triage meeting, but it's the sort of thing that's hard to really look through in a meeting, so we deferred it to async review.

@ehuss and I are looking at it now, on the @rust-lang/lang-docs side, as it is Reference-like material, as @RalfJung said. We agreed it'd be good clarify the bit raised in #140463 (comment) regarding provenance and transmute. As discussed in that thread, it'd probably be best to just describe the situation here, in terms of how the surrounding code effects a reborrow, and give an example or demonstration of code that's equivalent in this model.

When that's done, the typo I noted is fixed, and the text is wrapped more consistently, I'll r+ it.

Process-wise, thanks for pinging lang here; that was correct. In fact, let's plan to ping @rust-lang/lang and @rust-lang/lang-docs both on this sort of thing.

[rustbot]

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[traviscross]

I've revised the text for hopefully better clarity. In particular, as @RalfJung had suggested above, I've stated the guarantees upfront in positive form and then followed with the various caveats. This seemed desirable to me, and will hopefully ease lang review.

@RalfJung, @joshlf, does this look correct? (Did I break anything?)