Adjust matching rules videos

[phirestalker]

I have noticed that this software (like all others of its kind) have trouble with videos in low-light scenes. When there is not enough visual information in the scene, the algorithms seem to err towards false positives.

Some videos have an intro that will be matched to all the other videos from that producer.

Is there some way to tell the software to ignore scenes or frames below a certian brightnes threshold? For the other issue, I'm not sure what can be done, other than mark those as non-dulplicates. Is that possibe? Do you have any ideas other than mine how to solve these issues?

PS. This software does most of what I have been looking for. Most software of this kind will only match almost identical videos. yours has found clips of larger videos brilliantly. The only thing that could be better is a way to prefer the larger video and have some kind of saturation map to see if all the scenes are covered by the smaller clips (so they can be deleted).
Thanks for the great program. It has saved me a lot of time and space already.

[scrubbbbs]

Hey, thanks for the feedback.

Some videos have an intro that will be matched to all the other videos from that producer.

use p.vtrim to set the number of frames to ignore from the start/end

Is there some way to tell the software to ignore scenes or frames below a certian brightnes threshold?

This is being done to some extent but I don't have a setting for it.

I like the clip coverage idea, thanks for suggesting.

[phirestalker]

On a related note to the clip coverage. Can this software scan one of those screengrab collection images and match it to a video file? The images with multiple screengrabs every x minutes or whatever?

[scrubbbbs]

On a related note to the clip coverage. Can this software scan one of those screengrab collection images and match it to a video file? The images with multiple screengrabs every x minutes or whatever?

There is -select-grid for this. IIRC it requires some blank space between screengrabs.

cbird -select-grid screengrabs.png -p.alg video -similar-to @ -show

[phirestalker]

I was trying to figure out how to program something like this, and couldn't come up with a good method. Most of the ones I have seen from various tools do not put any space between the frames.
I toyed with using OCR to read the dimensions of the video from the header and detecting the end of the header. Then, dividing the remaining pixels using the found resolution, but that is messy at best. How did you end up implementing it?

[scrubbbbs]

It does need a flat or mostly flat background color between each frame. It finds the borders between thumbnails by looking for horizontal and vertical lines that span the image, which are defined as a series of pixels with close to the same brightness.

I think a similar strategy could be devised where you look for a line using edge-detection methods (but in reverse; i.e. you don't detect edges over the whole image, you trace a line and guess if it is an edge or not).

[phirestalker]

but in reverse; i.e. you don't detect edges over the whole image, you trace a line and guess if it is an edge or not

Yeah, I also wonder if you could start from the outside in and kind of like start with the largest squares and look for the smaller and smaller squares until you find ones where there's multiple squares of the same size.

You said the v trim setting was how many frames to skip in the beginning or end. So does your software use every frame in the video? Will I need to know the frame rate to kind of guess? I know most of the time the frame rate is 30 frames per second.

[scrubbbbs]

Yeah, I also wonder if you could start from the outside in and kind of like start with the largest squares and look for the smaller and smaller squares until you find ones where there's multiple squares of the same size.

I see what you are saying, but finding spanning lines/edges is way easier and should be sufficient as there is the assumption of a regular grid without too much margin/padding around it.

Otherwise you are building a rectangle detector with a constraint on it, something like a rect is only valid if it has enough edges on all sides, has a reasonable aspect ratio, and roughly repeats a whole number of times across/down the image. You could even do some statistics on on all the rects you found to select the most likely solution.

You said the v trim setting was how many frames to skip in the beginning or end. So does your software use every frame in the video? Will I need to know the frame rate to kind of guess? I know most of the time the frame rate is 30 frames per second.

It does decode every frame, but only stores frames that are not too similar to their neighbors. -i.vht sets the threshold for this. So you would need to know the frame rate to guess; most videos are 24-30 fps.

The video indexing is in need of an update to support longer videos and also things like this.

[phirestalker]

I was looking through the help, and there is a directory selection option. To me, it sounds like it is to select what to compare. How can I select the criteria for which image or video to keep. For instance, I want to keep one of a set of exact duplicates if the path has a certain word in it. Also, in general I find it better to keep the photos with more subdirectories, as they are more categorized. Are there any settings for that?

For near-duplicates I would want it to select the one with the highest resolution for a match above say 95 percent. It would be cool if it could move it to the location with the largest path as well.

Sorry, I never want to do anything simple. Thanks for all the help.

[scrubbbbs]

1. cbird -similar compares everything
2. cbird -select-*. -similar-to @ compares selected set to everything
3. cbird -select-* -similar-in @ compares selected set to itself

For instance, I want to keep one of a set of exact duplicates if the path has a certain word in it.

Maybe cbird -dups -without path '~word'

For near-duplicates I would want it to select the one with the highest resolution for a match above say 95 percent

I don't have a percentage scale but you can use a low threshold.

cbird -p.dht 1 -similar -sort-rev res -first

It would be cool if it could move it to the location with the largest path as well.

There is -move but it doesn't support any variable expansion on the destination path. Variable expansion doesn't operate on groups either.

[tehMArs]

Hey, thanks for the feedback.

Some videos have an intro that will be matched to all the other videos from that producer.

use p.vtrim to set the number of frames to ignore from the start/end

p.vtrim that option doesn't work correctly. Even when I use it, (-show) the blank screens on both sides still appear. I've tried various values, ranging from 100 to 3000, to skip the intro, but it doesn't make a difference. I've also noticed that some duplicate removal software has similar options, such as 'ignore white pixels' and 'ignore black pixels', which might be useful in this case.

[phirestalker]

I don't have a percentage scale but you can use a low threshold.

cbird -p.dht 1 -similar -sort-rev res -first

I like this one. The only thing that could make it better is making sure that the file with the largest resolution and file size is the first image. I can't find file size anywhere in prop. I wonder if it is available through ffmpeg metadata?

I have given up on the idea of keeping from a certain folder. The files are too disorganized for this to be useful. My ultimate goal for now is to remove all duplicates that are less quality with -nuke. The only problem is that -p.dht 1 finds images as matches where in one the subject is holding a glass and in the other they are not. In other software where they have a percent match, when it says 100% it can still be resized or recompressed, but it is always the same image. I was hoping -p.dht 0 would do that, but I get no matches with that one. Is there any way to find these simple resizes or recompressions without matching near duplicates?

EDIT: forgot to mention that I am using dct for database size. Do I need a more advanced algo to get what I want, or is it simply not possible yet?

I thought I saw quality score somewhere. What does that take into account? Is it the least compressed and largest resolution file by chance?

[scrubbbbs]

I like this one. The only thing that could make it better is making sure that the file with the largest resolution and file size is the first image. I can't find file size anywhere in prop. I wonder if it is available through ffmpeg metadata?

There is a fileSize property in the current git version. For multisort you can chain properties: -sort-rev res -sort-rev fileSize for example.

Is there any way to find these simple resizes or recompressions without matching near duplicates?

Nope, because they are technically all near duplicates. The only solution to your problem would be a hash code that had more resolution...cbirds is 64 bits but there is no reason it couldn't be wider (well besides the additional overhead).

There is one additional restriction you can add that might remove things you don't want to consider, the template matcher.

cbird -similar -p.tm 1 does a more detailed check for reasonable matches so it could reduce the number you have to go through. The other algos will match cropped and/or rotated images so they're not an option.

There is an attempt at quality scoring in the current version but it is very slow not very good (press "Q" on the UI and it will score the matches). It could be much better, but there is no fool-proof solution besides evaluating each match manually.

[phirestalker]

There is -select-grid for this. IIRC it requires some blank space between screengrabs.

cbird -select-grid screengrabs.png -p.alg video -similar-to @ -show

This gives me the error

@Commands::selectFiles$ select-files: file not found: "@"
@main$ similar-to: empty selection, is "@" a valid path or selector?

[scrubbbbs]

select-grid takes a file list. If the files don't exist it was doing odd things. I just committed a few changes to make it work like select-files. The syntax remains the same.

[phirestalker]

Are you saying it was not working because I only used one file?

[phirestalker]

About the other algos. I saw the size chart that you posted. I chose DCT and it works great. I have been manually sorting some images and come across ones that are resized and/or cropped. Some are only color adjusted, and they were not detected. I am thinking of using a more lax algo, but I need it to be still space efficient.

I figure I can make up for false positives in the more lax algo with the -p.tm 1 option you mentioned. So, of the other algos, which one would catch these resized/cropped/color adjusted ones and still be low space?

[scrubbbbs]

DCT is pretty good for resized and color adjusted but won't get them all.

For cropped images you need fdct (DCT of features) or orb (rotation-invariant features hash). Features are locations in the image that are rotation/scale/brightness invariant. Neither one uses that much space.

[phirestalker]

There is -select-grid for this. IIRC it requires some blank space between screengrabs.
cbird -select-grid screengrabs.png -p.alg video -similar-to @ -show

This gives me the error

@Commands::selectFiles$ select-files: file not found: "@"
@main$ similar-to: empty selection, is "@" a valid path or selector?

This is happening with a build from the newest commit. Should I open a new issue?

[scrubbbbs]

This is happening with a build from the newest commit. Should I open a new issue?

You probably already figured this out, but it looks like you used "@" as an argument to -select-grid, which is invalid for any of the -select-* options.

[phirestalker]

There is -select-grid for this. IIRC it requires some blank space between screengrabs.

cbird -select-grid screengrabs.png -p.alg video -similar-to @ -show

What should I use instead for your example?

[scrubbbbs]

In the current git version, to be more clear...

cbird -use <somedir> -select-grid [path1] ... [pathN] -p.alg video -similar-to @ -show

Where is any file or directory path. Note you can pass multiple paths separated by spaces if you like.

[phirestalker]

I cd to the directory with the video index. I then use the command
cbird -use . -select-grid screengrab.jpeg -p.alg video -similar-to @ -show
I still get the error

@Commands::selectFiles$ select-files: file not found: "@"
@main$ similar-to: empty selection, is "@" a valid path or selector?

What am I doing wrong?

EDIT: this is using a build from commit 2b24e1c

[scrubbbbs]

What am I doing wrong?

No idea. I compiled 2b24e1c and it works for me. Is that the complete output? I get something similar if the file screengrab.jpeg does not exist in the current directory or no grid was detected.. but there are more warnings before these two lines.

[phirestalker]

Full error.

(cbird:5138): dbind-WARNING **: 13:58:18.284: AT-SPI: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files

@Commands::selectFiles$ select-files: file not found: "@"
@main$ similar-to: empty selection, is "@" a valid path or selector?

Maybe I did something wrong when I compiled it?

[scrubbbbs]

Have you tried using -select-files <path> -show to see if that works? That command is doing the same thing as select-grid. Maybe has something to do with docker container. Does the current appimage work?

[phirestalker]

-select-files works with the same command line options and the same file. All I changed was that one option. This is odd.
It is a VM and not a container, and I built the program on the VM to ensure compatibility. I am unable to get an Appimage to compile, and there is not one for arm64 systems.
Oh, my system uses 16k pages instead of the more standard 4k. However, I would expect some serious whining or a full tantrum for errors related to memory.
So now we know it is not permissions or a problem with the file. What could be silently failing?

[phirestalker]

OK. It is the file. I looked through the code and the only place that could give up without an error is https://github.com/scrubbbbs/cbird/blob/41c4cc6235716211a8328d2dfd8c09b4cab63feb/src/cvutil.cpp#L1391C3-L1391C6
I then decided to find some more files to test with. One gave me that couldn't detect a grid message, and another failed without a message, but finally I got one that worked. Unfortunately, it was for a video I don't have. I will make my own screengrabs for a few videos and test with those.

EDIT: made my own screengrab and it matched to the video. Brilliant!
Also, one little nitpick. It is not able to detect the duplicate if the screengrab script uses a shadow in the output. The script uses montage with the shadow option.

[scrubbbbs]

I have to admire your persistence, good job!

The source line you found (if it was the branch taken) could mean the image format is unsupported. Maybe it is not what it seems, (wrong extension?).

Also, one little nitpick. It is not able to detect the duplicate if the screengrab script uses a shadow in the output. The script uses montage with the shadow option.

It makes sense that would be a problem. If there is too much brightness variation between pixels in the line detector then it won't be considered a valid line. There are other methods which we discussed earlier, maybe using hue instead of brightness for example could work around this particular issue.