sliding frame

[camhart]

I'd love to be able to add support for a sliding frame. I know I could do this outside the library, by cropping the image multiple times, or potentially at the buffer level. My gut says the performance would be significantly improved if I pushed it down into tensorflow though (is that possible? Is https://js.tensorflow.org/api/latest/#signal.frame the right way to do it?).

I'm willing to put some leg-work in--but I'd really appreciate some guidance or if anyone has resources describing how to do it with a tensorflow.js implementation.

[GantMan]

You're right! I think Signal frame is the way to go.

What do you foresee as the API?

You call a frame based function and then pass the params for the frames? Would it then average the frames? Or give a report back like the GIF implementation?

I think sliding frame would be an excellent addition!

[camhart]

Well--with my limited understanding--I'd want to be able to define the window--it's size and how it moves. I have no idea if this is the ideal way to handle this... but passing in an array of "sliding window definitions" like so:

 [{ width: 300, height: 300, moveX: 250, moveY: 250}, { width: 800, height: 800, moveX: 600, moveY: 600}]

width - defines the width of the window
height - defines the height of the window
moveX - determines how far the window is moved in the x axis when it "slides" over
moveY - determines how far the window is moved in the y axis when it "slides" down

I think an array would be ideal because if you're dealing with large resolution images you could pick up on small and large portions of the screen as "nsfw". You could make the array optional and generate it if not provided based off the input image's width/height.

I'll need some more time to research how to tie in Signal Frame--if you have any thoughts please share.

One thing I'm not sure about--how do I go from a tf.Tensor3D (which is what img is in infer), to then tf.Tensor1D which is what tf.signal.frame uses? It looks like tf.split might be what I want.

Also--why is img 3D instead of 2D?

[GantMan]

Images are 3D bc widthxheightxRGB

I'll try to circle back on sliding frame at some point.