Mapping pyhf's full likelihoods to the simplified likelihood framework

[jackaraz]

Hi all, I'm adding this here in case it interests others. During the last (Re)interpretation workshop, we discussed a possible way to map pyhf full likelihoods to the simplified likelihood framework. This is especially important for cases where the full likelihood is highly expensive. After discussing with Nick Wardle (@nucleosynthesis), Sabine Kraml (@sabinekraml) and Wolfgang Waltenberger (@WolfgangWaltenberger), I implemented a plug-in for Spey to handle this conversion (can be found in the latest release). The details on the methodology and usage can be found in this link.

In summary, this method mainly compresses the nuisance parameters into a single uncertainty source per bin by creating a multivariate Gaussian with the covariance matrix of the nuisance parameters. Then, we sample from the original model using the nuisance parameters generated by this multivariate Gaussian and estimate the correlations between background bins. @nucleosynthesis tells me that they use this method for CMS' simplified likelihoods. I can reasonably reconstruct the exclusion limits for the ATLAS-SUSY-2019-08 analysis. Still, it is essential to note that this very much depends on how you construct the multivariate Gaussian for the nuisance parameters. Hence, I provided all the possible options that can be manipulated up to the user.

I hope it's useful!
Cheers

[kratsg]

Hi Jack, you are aware of this work by (primarily) former PhD Student Eric Schanet right? https://pypi.org/project/simplify/ -- there's lots of ways to build simplified likelihoods. It would be good to have a separate thing that demonstrates various ways of building simplified likelihoods, each with their tradeoffs.

[jackaraz]

Hi @kratsg, yes, I'm aware that we used Eric's code in this paper. I can confidently say that Eric's code is much faster to create a working simplified likelihood since there is no sampling. But of course, there are various ways to simplify a likelihood, as you say, so it would indeed be great to create a pros and cons chart. However, I am unsure what would be a valuable, quantifiable measure to compare different approaches.

[kratsg]

This is probably something I'd like to see a proper discussion on at a future RiF workshop perhaps. I think it's valuable to have sort of a whitepaper including stat experts that describe the various approaches that could be taken, pros and cons, computational complexity, etc etc etc. The main concern from my side is that a lot of these developments will get based off pyhf (ok, fine, that's a good problem?) because the HiFa JSON is a lot more approachable to play with than anything else that exists right now -- but it might limit the usability of these methods if they're not generalized enough to extend beyond what I imagine is a very limited scope of a subset of binned statistical models.

Perhaps what we can do for our pyhf documentation is provide a page that summarizes additional tools to use in the "family" such as cabinetry, simplify, spey.simplify, etc...

[jackaraz]

This is probably something I'd like to see a proper discussion on at a future RiF workshop perhaps. I think it's valuable to have sort of a whitepaper including stat experts that describe the various approaches that could be taken, pros and cons, computational complexity, etc etc etc.

Noted!

The main concern from my side is that a lot of these developments will get based off pyhf (ok, fine, that's a good problem?) because the HiFa JSON is a lot more approachable to play with than anything else that exists right now -- but it might limit the usability of these methods if they're not generalized enough to extend beyond what I imagine is a very limited scope of a subset of binned statistical models.

It's absolutely not a problem; pyhf is a great tool, but if it did everything, it would be a nightmare to debug the beast. :D

That's a great point and one of the main reasons why I developed spey as a plug-in-based interface. In principle, likelihood structure can be anything binned, unbinned, ML-based, you name it (also the fact that I don't know all types of likelihoods one might want to construct); one should be able to use any of these to test hypotheses. So, the idea behind spey is: you tell me how this likelihood should be computed, and I'll let you know if your model is excluded or I'll fit your parameters etc. etc. And one can publish their custom likelihood structure to be used in other studies.

Perhaps what we can do for our pyhf documentation is provide a page that summarizes additional tools to use in the "family" such as cabinetry, simplify, spey.simplify, etc...

That sounds great!

[kratsg]

I guess spey is the implementation (on some level) of what HS3 is the specification for.

[jackaraz]

I believe yes, the goal is the same where the origin of likelihood can be anything or any already publicly available software such as pyhf, fastprof or CMS' combine in the future, where you can write a simple plug-in and spey can automatically detect this plug-in once its installed. But I ofc, can not claim that it can do everything since this was my attempt to learn some statistics and unite the statistics interface for all reinterpretation tools so that one can combine the statistical models of different searches/measurements/experiments together no matter how the statistical model is described. The use cases will evolve hopefully, as I learn but ofc if you have any suggestions its more than welcome!