file_reporter argument to RunSpecifiedBenchmarks should be optional, but is not

[tlang0]

In the comment/documentation of RunSpecifiedBenchmarks in benchmark.h, it says:

// The second and third overload use the specified 'display_reporter' and
// 'file_reporter' respectively. 'file_reporter' will write to the file
// specified
// by '--benchmark_output'. If '--benchmark_output' is not given the
// 'file_reporter' is ignored.

The first problem is that the correct argument name is benchmark_out (I assume it was changed at some point).
The second problem is that the last sentence is not true. If ' --benchmark_out' is not given, the following message is displayed:

A custom file reporter was provided but --benchmark_out= was not specified.

I don't know which is the intended behavior, but to me, it would make sense that the file_reporter is ignored if '--benchmark_out' is not specified.

[dmah42]

The documentation does tend to lag. PRs welcome :) For the second issue, i'd take it as read that the behaviour of the code is the desired behaviour. If you want to debate that it should be the alternative then go for it, but the idea is that if you've gone to the effort of providing us with a file reporter, you probably meant to use it (and this error avoids the obvious bug report). A third option would be to take it as given that if a custom file reporter is provided we should use it (ie, implicitly set benchmark_out to some default value). that may be surprising?

…

On Fri, Jul 12, 2019 at 12:58 PM Thomas Lang ***@***.***> wrote: In the comment/documentation of RunSpecifiedBenchmarks in benchmark.h, it says: // The second and third overload use the specified 'display_reporter' and // 'file_reporter' respectively. 'file_reporter' will write to the file // specified // by '--benchmark_output'. If '--benchmark_output' is not given the // 'file_reporter' is ignored. The first problem is that the correct argument name is benchmark_out (I assume it was changed at some point). The second problem is that the last sentence is not true. If ' --benchmark_out' is not given, the following message is displayed: A custom file reporter was provided but --benchmark_out= was not specified. I don't know which is the intended behaviour, but to me, it would make sense that the file_reporter is ignored if '--benchmark_out' is not specified. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#836?email_source=notifications&email_token=AAD4QMVK2LV2DZAMG4D5DF3P7BWYFA5CNFSM4ICIKNG2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G63S7EQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD4QMTWYEJMKHWRJPF463LP7BWYFANCNFSM4ICIKNGQ> .

[tlang0]

Thank you for your answer and sorry for my late response.

My intention was to provide a custom JSONReporter which is used only if the benchmark is executed with the --benchmark_out parameter pointing to a JSON output file. Otherwise, there would just be the console output. This way, the user can decide if they want to have the additional JSON output written to a file or just the console output.

This is exactly how it works if no custom BenchmarkReporter is provided. If the --benchmark_out argument is set, the results are written to the specified file using the format specified by --benchmark_out_format. Otherwise, only the console output is produced.

If a custom BenchmarkReporter is provided as file_reporter argument to RunSpecifiedBenchmarks, the behavior changes. Now, it is no longer possible to execute the benchmark without the --benchmark_out parameter. It only allows writing to a file. This means that with the custom file_reporter, the user can no longer decide if they want to see the output directly or if it should be written to a file.

This difference in behavior is why I thought that it is probably an error (not just in the documentation, but in the behavior). But I might just have overlooked something or misunderstood the function parameters?