Skip to content

Commit

Permalink
Expose additional information about decoded frames in the viewer (rer…
Browse files Browse the repository at this point in the history 
…un-io#7932)

### What


![image](https://github.com/user-attachments/assets/6d92bab1-107f-4026-9e13-fa87a2cc122b)

Minimized by default.

Originally, I wanted to add some more information from the get-go like
i/idr/p/b-frame type but turns out we don't really have all that much
information around today for sure - a big part of this is that typically
just pass in chunks to decoder and get 0-1 frames out for each chunk (my
current understand is that it could be 0-n, but n>1 typically doesn't
make sense because a single mp4 sample has a single timestamp!). Meaning
we typically don't maintain the chunk/frame relationship.
However, this adjusted structure should make it easy to expose optional
fields on `FrameInfo` to forward all sort of things

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7932?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7932?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!
* [x] If have noted any breaking changes to the log API in
`CHANGELOG.md` and the migration guide

- [PR Build Summary](https://build.rerun.io/pr/7932)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

To run all checks from `main`, comment on the PR with `@rerun-bot
full-check`.
  • Loading branch information
@Wumpf
Wumpf authored Oct 30, 2024
1 parent 97e944b commit 413f7a1
Show file tree
Hide file tree
Showing 12 changed files with 417 additions and 240 deletions.
7 changes: 6 additions & 1 deletion crates/store/re_video/examples/frames.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,12 @@ fn main() {
.truncate(true)
.open(output_dir.join(format!("{i:0width$}.ppm")))
.expect("failed to open file");
write_binary_ppm(&mut file, frame.width, frame.height, &frame.data);
write_binary_ppm(
&mut file,
frame.content.width,
frame.content.height,
&frame.content.data,
);
}
}
}
Expand Down
20 changes: 12 additions & 8 deletions crates/store/re_video/src/decode/av1.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ use crate::Time;
use dav1d::{PixelLayout, PlanarImageComponent};

use super::{
async_decoder_wrapper::SyncDecoder, Chunk, Error, Frame, OutputCallback, PixelFormat, Result,
YuvMatrixCoefficients, YuvPixelLayout, YuvRange,
async_decoder_wrapper::SyncDecoder, Chunk, Error, Frame, FrameContent, FrameInfo,
OutputCallback, PixelFormat, Result, YuvMatrixCoefficients, YuvPixelLayout, YuvRange,
};

pub struct SyncDav1dDecoder {
Expand Down Expand Up @@ -228,12 +228,16 @@ fn output_picture(
};

let frame = Frame {
data,
width: picture.width(),
height: picture.height(),
format,
presentation_timestamp: Time(picture.timestamp().unwrap_or(0)),
duration: Time(picture.duration()),
content: FrameContent {
data,
width: picture.width(),
height: picture.height(),
format,
},
info: FrameInfo {
presentation_timestamp: Time(picture.timestamp().unwrap_or(0)),
duration: Time(picture.duration()),
},
};
on_output(Ok(frame));
}
Expand Down
54 changes: 45 additions & 9 deletions crates/store/re_video/src/decode/mod.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -198,23 +198,59 @@ pub struct Chunk {
pub duration: Time,
}

/// Data for a decoded frame on native targets.
#[cfg(not(target_arch = "wasm32"))]
pub type FrameData = Vec<u8>;

#[cfg(target_arch = "wasm32")]
pub type FrameData = webcodecs::WebVideoFrame;

/// One decoded video frame.
pub struct Frame {
pub data: FrameData,
pub struct FrameContent {
pub data: Vec<u8>,
pub width: u32,
pub height: u32,
pub format: PixelFormat,
}

/// Data for a decoded frame on the web.
#[cfg(target_arch = "wasm32")]
pub type FrameContent = webcodecs::WebVideoFrame;

/// Meta information about a decoded video frame, as reported by the decoder.
#[derive(Debug, Clone)]
pub struct FrameInfo {
/// The presentation timestamp of the frame.
///
/// Decoders are required to report this.
/// A timestamp of [`Time::MAX`] indicates that the frame is invalid or not yet available.
pub presentation_timestamp: Time,

/// How long the frame is valid.
///
/// Decoders are required to report this.
/// A duration of [`Time::MAX`] indicates that the frame is invalid or not yet available.
// Implementation note: unlike with presentation timestamp we may be able fine with making this optional.
pub duration: Time,
}

/// Pixel format/layout used by [`Frame::data`].
impl Default for FrameInfo {
fn default() -> Self {
Self {
presentation_timestamp: Time::MAX,
duration: Time::MAX,
}
}
}

impl FrameInfo {
/// Presentation timestamp range in which this frame is valid.
pub fn time_range(&self) -> std::ops::Range<Time> {
self.presentation_timestamp..self.presentation_timestamp + self.duration
}
}

/// One decoded video frame.
pub struct Frame {
pub content: FrameContent,
pub info: FrameInfo,
}

/// Pixel format/layout used by [`FrameContent::data`].
#[derive(Debug)]
pub enum PixelFormat {
Rgb8Unorm,
Expand Down
17 changes: 6 additions & 11 deletions crates/store/re_video/src/decode/webcodecs.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ use web_sys::{
};

use super::{
AsyncDecoder, Chunk, DecodeHardwareAcceleration, Frame, OutputCallback, PixelFormat, Result,
AsyncDecoder, Chunk, DecodeHardwareAcceleration, Frame, FrameInfo, OutputCallback, Result,
};
use crate::{Config, Time, Timescale};

Expand Down Expand Up @@ -179,17 +179,12 @@ fn init_video_decoder(
Time::from_micros(frame.timestamp().unwrap_or(0.0), timescale);
let duration = Time::from_micros(frame.duration().unwrap_or(0.0), timescale);

let frame = WebVideoFrame(frame);
let width = frame.display_width();
let height = frame.display_height();

on_output(Ok(Frame {
data: frame,
width,
height,
presentation_timestamp,
format: PixelFormat::Rgba8Unorm,
duration,
content: WebVideoFrame(frame),
info: FrameInfo {
presentation_timestamp,
duration,
},
}));
}) as Box<dyn Fn(web_sys::VideoFrame)>)
};
Expand Down
175 changes: 1 addition & 174 deletions crates/viewer/re_data_ui/src/blob.rs
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
use re_renderer::{external::re_video::VideoLoadError, video::VideoFrameTexture};
use re_types::components::{Blob, MediaType, VideoTimestamp};
use re_ui::{list_item::PropertyContent, UiExt};
use re_viewer_context::UiLayout;

use crate::{image::image_preview_ui, EntityDataUi};
use crate::{image::image_preview_ui, video::show_video_blob_info, EntityDataUi};

impl EntityDataUi for Blob {
fn entity_data_ui(
Expand Down Expand Up @@ -169,175 +168,3 @@ pub fn blob_preview_and_save_ui(
});
}
}

fn show_video_blob_info(
render_ctx: Option<&re_renderer::RenderContext>,
ui: &mut egui::Ui,
ui_layout: UiLayout,
video_result: &Result<re_renderer::video::Video, VideoLoadError>,
video_timestamp: Option<VideoTimestamp>,
blob: &re_types::datatypes::Blob,
) {
#[allow(clippy::match_same_arms)]
match video_result {
Ok(video) => {
if ui_layout.is_single_line() {
return;
}

let data = video.data();

re_ui::list_item::list_item_scope(ui, "video_blob_info", |ui| {
ui.list_item_flat_noninteractive(
PropertyContent::new("Dimensions").value_text(format!(
"{}x{}",
data.width(),
data.height()
)),
);
if let Some(bit_depth) = data.config.stsd.contents.bit_depth() {
let mut bit_depth = bit_depth.to_string();
if data.is_monochrome() == Some(true) {
bit_depth = format!("{bit_depth} (monochrome)");
}

ui.list_item_flat_noninteractive(
PropertyContent::new("Bit depth").value_text(bit_depth),
);
}
if let Some(subsampling_mode) = data.subsampling_mode() {
// Don't show subsampling mode for monochrome, doesn't make sense usually.
if data.is_monochrome() != Some(true) {
ui.list_item_flat_noninteractive(
PropertyContent::new("Subsampling mode")
.value_text(subsampling_mode.to_string()),
);
}
}
ui.list_item_flat_noninteractive(
PropertyContent::new("Duration")
.value_text(format!("{}", re_log_types::Duration::from(data.duration()))),
);
// Some people may think that num_frames / duration = fps, but that's not true, videos may have variable frame rate.
// At the same time, we don't want to overload users with video codec/container specific stuff that they have to understand,
// and for all intents and purposes one sample = one frame.
// So the compromise is that we truthfully show the number of *samples* here and don't talk about frames.
ui.list_item_flat_noninteractive(
PropertyContent::new("Sample count")
.value_text(re_format::format_uint(data.num_samples())),
);
ui.list_item_flat_noninteractive(
PropertyContent::new("Codec").value_text(data.human_readable_codec_string()),
);

if ui_layout != UiLayout::Tooltip {
ui.list_item_collapsible_noninteractive_label("MP4 tracks", true, |ui| {
for (track_id, track_kind) in &data.mp4_tracks {
let track_kind_string = match track_kind {
Some(re_video::TrackKind::Audio) => "audio",
Some(re_video::TrackKind::Subtitle) => "subtitle",
Some(re_video::TrackKind::Video) => "video",
None => "unknown",
};
ui.list_item_flat_noninteractive(
PropertyContent::new(format!("Track {track_id}"))
.value_text(track_kind_string),
);
}
});
}

if let Some(render_ctx) = render_ctx {
// Show a mini-player for the video:

let timestamp_in_seconds = if let Some(video_timestamp) = video_timestamp {
video_timestamp.as_seconds()
} else {
// TODO(emilk): Some time controls would be nice,
// but the point here is not to have a nice viewer,
// but to show the user what they have selected
ui.ctx().request_repaint(); // TODO(emilk): schedule a repaint just in time for the next frame of video
ui.input(|i| i.time) % video.data().duration().as_secs_f64()
};

let player_stream_id = re_renderer::video::VideoPlayerStreamId(
ui.id().with("video_player").value(),
);

match video.frame_at(
render_ctx,
player_stream_id,
timestamp_in_seconds,
blob.as_slice(),
) {
Ok(VideoFrameTexture {
texture,
time_range,
is_pending,
show_spinner,
}) => {
let response = crate::image::texture_preview_ui(
render_ctx,
ui,
ui_layout,
"video_preview",
re_renderer::renderer::ColormappedTexture::from_unorm_rgba(texture),
);

if is_pending {
ui.ctx().request_repaint(); // Keep polling for an up-to-date texture
}

if show_spinner {
// Shrink slightly:
let smaller_rect = egui::Rect::from_center_size(
response.rect.center(),
0.75 * response.rect.size(),
);
egui::Spinner::new().paint_at(ui, smaller_rect);
}

response.on_hover_ui(|ui| {
// Prevent `Area` auto-sizing from shrinking tooltips with dynamic content.
// See https://github.com/emilk/egui/issues/5167
ui.set_max_width(ui.spacing().tooltip_width);

let timescale = video.data().timescale;
ui.label(format!(
"Frame at {} - {}",
re_format::format_timestamp_seconds(
time_range.start.into_secs(timescale),
),
re_format::format_timestamp_seconds(
time_range.end.into_secs(timescale),
),
));
});
}

Err(err) => {
ui.error_label_long(&err.to_string());
}
}
}
});
}
Err(VideoLoadError::MimeTypeIsNotAVideo { .. }) => {
// Don't show an error if this wasn't a video in the first place.
// Unfortunately we can't easily detect here if the Blob was _supposed_ to be a video, for that we'd need tagged components!
// (User may have confidently logged a non-video format as Video, we should tell them that!)
}
Err(VideoLoadError::UnrecognizedMimeType) => {
// If we couldn't detect the media type,
// we can't show an error for unrecognized formats since maybe this wasn't a video to begin with.
// See also `MediaTypeIsNotAVideo` case above.
}
Err(err) => {
if ui_layout.is_single_line() {
ui.error_label(&format!("Failed to load video: {err}"));
} else {
ui.error_label_long(&format!("Failed to load video: {err}"));
}
}
}
}
1 change: 1 addition & 0 deletions crates/viewer/re_data_ui/src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ mod image;
mod instance_path;
mod store_id;
mod tensor;
mod video;

pub mod item_ui;

Expand Down
Loading

0 comments on commit 413f7a1

Please sign in to comment.