Add PyTorch Profiler to track performance

[gomezzz]

Feature

Desired Behavior / Functionality

Until now we have used cprofile to investigate performance of torchquad. However, especially for GPUs it may be interesting to start investigating with the new PyTorch profiler. Especially, as it has recently received some upgrades.

What Needs to Be Done

• Set up things so that profiling with PyTorch profiler is easily doable
• Profile!
• Feel free to sum up your conclusions as to what optimization targets might be from this.
• (optionally) add a notebook or some code to run a performance analysis of torchquad using the profiler.

[gomezzz]

@FHof Do you think this issue still makes sense? Otherwise, I would close it :)

[FHof]

I think it only makes sense if you want to have the profiling code in this repository.
The profiling and benchmarking code which I have used are all in that folder of my fork: https://github.com/FHof/torchquad/tree/fritz/autoray/benchmarking_and_profiling
(Here's a copy of it in case the above link stopped working: benchmarking_and_profiling.zip)
The profiling code is in do_profile.py and a generated output for JAX, TensorFlow and PyTorch can be viewed in TensorBoard.
Example output for the profilers can be seen at FHof#13 and in my thesis.

[gomezzz]

Perfect thanks! I will close this for now then.