Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update memtrace analyzer docs and tools for skipped headers when fast-forwarding #7224

Open
abhinav92003 opened this issue Jan 27, 2025 · 2 comments
Assignees

Comments

@abhinav92003
Copy link
Contributor

The drmemtrace scheduler allows the user to specify regions of interest in the trace.

std::vector<range_t> regions_of_interest;

As documented, when this is specified, entries outside these regions are simply skipped. What is not documented explicitly, is that this may include the trace header entries, such as TRACE_MARKER_TYPE_FILETYPE.

Some trace analysis tools rely on seeing the TRACE_MARKER_TYPE_FILETYPE marker and fail if they don't see it, particularly opcode_mix. E.g.,

shard->error = "No file type found in this shard";
.

This issue is to fix such tools and also improve documentation.

@abhinav92003
Copy link
Contributor Author

Though another way to look at this is that -skip_instrs asks for instrs to be skipped, not memrefs. And I think scheduler region_of_interests apply only to instrs, not memrefs. So maybe another possibility is that we provide the header entries regardless of the scheduler regions of interest?

@derekbruening
Copy link
Contributor

We discussed this while implementing the skip-instruction feature #5538 and decided to not duplicate any headers and to instead add stream API access in PR #5739 . The discussion included this:


** DONE should top headers in each thread be re-emitted at seek point if skip them (b/c start from 0, or b/c using serial or per-cpu iterator)? => no: user has to do linear walk if wants to read them (but reader has to do this too to get chunk size)

Should the top headers be cached and re-emitted at the seek point if
seeking from 0? Or, require analyzer or external iterator to walk the
headers first before skipping if they want those values? What about top
headers in other threads in serial or per-cpu iteration? Headers in other
threads are pretty much the same so maybe no point in worrying about this:

Output format:
<record#> <instr#>: T<tid> <record details>
------------------------------------------------------------
        1        0: T2012127 <marker: version 3>
        2        0: T2012127 <marker: filetype 0x40>
        3        0: T2012127 <marker: cache line size 64>
        4        0: T2012127 <marker: chunk instruction count 10000000>
        5        0: T2012127 <marker: page size 4096>
------------------------------------------------------------
        6        0: T2012128 <marker: version 3>
        7        0: T2012128 <marker: filetype 0x40>
        8        0: T2012128 <marker: cache line size 64>
        9        0: T2012128 <marker: chunk instruction count 10000000>
       10        0: T2012128 <marker: page size 4096>
------------------------------------------------------------
       11        0: T2012124 <marker: version 3>
       12        0: T2012124 <marker: filetype 0x40>
       13        0: T2012124 <marker: cache line size 64>
       14        0: T2012124 <marker: chunk instruction count 10000000>
       15        0: T2012124 <marker: page size 4096>
------------------------------------------------------------
       16        0: T2012126 <marker: version 3>
       17        0: T2012126 <marker: filetype 0x40>
       18        0: T2012126 <marker: cache line size 64>
       19        0: T2012126 <marker: chunk instruction count 10000000>
       20        0: T2012126 <marker: page size 4096>
------------------------------------------------------------
       21        0: T2012125 <marker: version 3>
       22        0: T2012125 <marker: filetype 0x40>
       23        0: T2012125 <marker: cache line size 64>
       24        0: T2012125 <marker: chunk instruction count 10000000>
       25        0: T2012125 <marker: page size 4096>
------------------------------------------------------------
       26        0: T2012124 <marker: timestamp 13307132907343045>
       27        0: T2012124 <marker: tid 2012124 on core 8>
       28        1: T2012124 ifetch       3 byte(s) @ 0x00007f2825745050 48 89 e7             mov    %rsp, %rdi

=>
Seems reasonable to require any iterator to look for headers before skipping;
not worth spending effort to re-emit them at skip point if skip from 0 in
that thread, or to re-emit in other threads.

Or, reader caches top headers and provides API to query.


@derekbruening derekbruening changed the title Scheduler ROI settings may skip over trace header entries Update memtrace analyzer docs and tools for skipped headers when fast-forwarding Jan 27, 2025
abhinav92003 added a commit that referenced this issue Jan 27, 2025
Fixes the opcode_mix tool to get the filetype from the memtrace_stream_t interface instead
of the TRACE_MARKER_TYPE_FILETYPE marker memref which may not be seen if -skip_instrs is
applied by the user.

Adds a new unit test that runs the opcode_mix tool with -skip_instrs, which would fail
because of an unseen filetype prior to this fix.

Issue: #7224, #7113
derekbruening added a commit that referenced this issue Jan 28, 2025
Clarifies that the -{skip,sim,warmup}_refs options skip a count of
trace records inside particular tools, rather than in the framework.

Clarifies that the -skip_{instrs,to_timestamp} options skip over
top-level markers, but their values are available from the stream API.

Adds i#7230 TODO comments on fixing the -{sim,warmup}_refs options in
the simulators to count marker records (or possibly to keep as is and
update the docs).

Issue: #7224, #7230
abhinav92003 added a commit that referenced this issue Jan 28, 2025
Fixes the opcode_mix tool to get the filetype from the memtrace_stream_t
interface instead of the TRACE_MARKER_TYPE_FILETYPE marker memref which
may not be seen if -skip_instrs is applied by the user.

Modifies opcode_mix_t to use initialize_stream instead of initialize, to
get the serial stream.

Adds a new unit test that runs the opcode_mix tool with -skip_instrs,
which would fail because of an unseen filetype prior to this fix.

Augments decode_cache_t::init() documentation to suggest a reliable way
to obtain the filetype.

Issue: #7224, #7113
abhinav92003 added a commit that referenced this issue Jan 30, 2025
Refactors the view_t tool to use the new decode_cache_t instead of
duplicating the instr decoding logic.

Modifies decode_info_base_t::set_decode_info_derived to allow it to
return the error string. This is useful for cases where there was an
error during its operation, which is especially possible when used with
a decode_cache_t with include_decoded_instr_ = false in which case the
decode_info_t also performs the decoding on its own.

Adds new tests to decode_cache_test to verify some related cases. Not
adding a new release note for the API signature change as there was no
intervening release since it was initially added.

Also fixes the view_t logic to get filetype, which should be obtained
from the memtrace_stream_t object instead of TRACE_MARKER_TYPE_FILETYPE
to handle cases where the filetype marker is not seen by the tool
because of analyzer options like -skip_instrs that skip over a certain
initial part of the trace.

Issue: #7113, #7224
derekbruening added a commit that referenced this issue Jan 30, 2025
Clarifies that the -{skip,sim,warmup}_refs options skip a count of trace
records inside particular tools, rather than in the framework.

Clarifies that the -skip_{instrs,to_timestamp} options skip over
top-level markers, but their values are available from the stream API.

Adds i#7230 TODO comments on fixing the -{sim,warmup}_refs options in
the simulators to count marker records (or possibly to keep as is and
update the docs).

Issue: #7224, #7230
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants