[jekoorn]

Closes #328

n3lo_cf_variation seems to be a single number, although I believe it should be 4 numbers: (C2g, C2q, CLg, CLq), i.e. one per coefficient we want to vary

[jekoorn]

currently it looks like the light coefficient functions support no variation at all, e.g. looking at https://github.com/NNPDF/yadism/blob/master/src/yadism/coefficient_functions/light/kernels.py#L78 vs https://github.com/NNPDF/yadism/blob/master/src/yadism/coefficient_functions/heavy/kernels.py#L76 . do we want to implement this as well at the same time?

[juanrojochacon]

currently it looks like the light coefficient functions support no variation at all, e.g. looking at https://github.com/NNPDF/yadism/blob/master/src/yadism/coefficient_functions/light/kernels.py#L78 vs https://github.com/NNPDF/yadism/blob/master/src/yadism/coefficient_functions/heavy/kernels.py#L76 . do we want to implement this as well at the same time?

@jekoorn the light coefficient functions at N3LO are known exactly so no, they should not be varied in any way. IHOUs are only required for the N3LO massive coefficient functions

[jekoorn]

How does the code distinguish between whether an FL or F2 component is called? I don't really see how I should tell the code to use C2g vs CLg

[juanrojochacon]

Felix will confirm, but I think that the distinction is made through

kind = esf.info.obs_name.kind at the top of generate, which is then used in import_pc_module(kind, esf.process). So indeed it is ok that you don't see the distinction there

[felixhekhorn]

Please remember to resolve the conversations, which you believe to have addressed

[felixhekhorn]

Also please don't code only in the void - at some point you need to actually run locally yourself! this is to say: you still have syntax errors

[jekoorn]

Also please don't code only in the void

yeah sorry for the mess, my editor didn't catch the syntax errors for some reason. I am apparently rusty after the holidays🙃

Another question: Do we only require a variation in the case that we evaluate F2 and FL? Not the others? (Are they known already?)
And if so, I think with the if/else statements it's a bit messy. But it looks like the other components don't use it, so it doesn't matter what we pass.

[juanrojochacon]

yes, only F2 and FL, xF3 is not relevant here

[felixhekhorn]

Let me ask again: have you actually run this code? and check e.g. that when you put different numbers this is propagated down to adani correctly?

[jekoorn]

have you actually run this code?

I am currently checking this with pinefarm (since I have to generate grids anyway and needed a pipeline). But probably it's easier to do this with yadism standalone 🙈

[jekoorn]

There are still some NaNs spawning, apparently. I will investigate this..
A fresh yadism also gets NaNs though, so I am not sure to what extent the NaNs are "supposed" to be there or not..

[Radonirinaunimi]

There are still some NaNs spawning, apparently. I will investigate this.. A fresh yadism also gets NaNs though, so I am not sure to what extent the NaNs are "supposed" to be there or not..

Could you please provide more details (which cards are you using, do the NaNs appear for specific processes/datasets/variations, etc.)? There are some NaNs that are fine (such as at very low $Q^2$) and there are some that are dangerous. From the recent grid computations, the latter should no longer appear.

[jekoorn]

Could you please provide more details (which cards are you using, do the NaNs appear for specific processes/datasets/variations, etc.)?

Sure!
For checking whether the variations produce different outputs, instead of using cards, I am using my own script that computes the structure functions. Just electron-proton collision at 1TeV.
I am integrating $Q^2$ from as low as 2.0, and x > 1e-6. I am using the theory cards from NNPDF/nnpdf#2494 but with the CF variations put in.
The broader issue is here that the outputs from different variations still yield the exact same output. I am wondering now if the setup I use for checking this is actually sufficient.
See https://vp.nnpdf.science/nVd3gBnfT2S-psG9B5g5gQ==/ for the script
Probably a good check would be whether running with actual pinecards yields NaNs, but for now I wanted to check whether the CF variations do what they are supposed to.

[felixhekhorn]

The broader issue is here that the outputs from different variations still yield the exact same output.

I suggest to put a debugger (or print or whatever) here

yadism/src/yadism/coefficient_functions/heavy/f2_nc.py

Line 49 in 467615f

"C2g", nf=self.nf, variation=self.n3lo_cf_variation

and check what variation arrives there and then what you get back

Also do you get finite numbers with the default settings? i.e. without any variation? so we know the problem is with the variations (and not with something else)

[jekoorn]

put a debugger (or print or whatever) here

nothing looks to arrive there, with whatever variation I put in. Probably explains why the calculations yield the same result between different variations. It makes me doubt whether the correlated variations were even doing what they should in the first place, before starting this implementation.

With default settings I also get NaNs, but to be precise: they don't appear in the output, it's just a notification that the runner passes:

 took 6.43 s
[14:03:10] CRITICAL Some NaNs are encountered and set to zero!                                                                                                                                                         runner.py:224
           CRITICAL Some NaNs are encountered and set to zero!                                                                                                                                                         runner.py:224
           CRITICAL Some NaNs are encountered and set to zero!                                                                                                                                                         runner.py:224
           CRITICAL Some NaNs are encountered and set to zero!                                                                                                                                                         runner.py:224
           CRITICAL Some NaNs are encountered and set to zero!

[Radonirinaunimi]

put a debugger (or print or whatever) here

nothing looks to arrive there, with whatever variation I put in. Probably explains why the calculations yield the same result between different variations. It makes me doubt whether the correlated variations were even doing what they should in the first place, before starting this implementation.

With default settings I also get NaNs, but to be precise: they don't appear in the output, it's just a notification that the runner passes:

 took 6.43 s
[14:03:10] CRITICAL Some NaNs are encountered and set to zero!                                                                                                                                                         runner.py:224
           CRITICAL Some NaNs are encountered and set to zero!                                                                                                                                                         runner.py:224
           CRITICAL Some NaNs are encountered and set to zero!                                                                                                                                                         runner.py:224
           CRITICAL Some NaNs are encountered and set to zero!                                                                                                                                                         runner.py:224
           CRITICAL Some NaNs are encountered and set to zero!

These NaNs are fine (as I per above) so you shouldn't worry about those. They are in the regions where pQCD breaks down and these warnings are just there warn people that these regions are probed.

[felixhekhorn]

They are in the regions where pQCD breaks down and these warnings are just there warn people that these regions are probed.

@Radonirinaunimi I think you put these warnings there in the first place, right? so you know that they are in this region? because this can not be inferred from the output ... (i.e. if we have the kinematics in hand there it might be good to echo them alongside 🙃 )

nothing looks to arrive there, with whatever variation I put in. Probably explains why the calculations yield the same result between different variations. It makes me doubt whether the correlated variations were even doing what they should in the first place, before starting this implementation.

@jekoorn then a) you should check on master what you get there (and confirm or reject your doubts) and b) then you "just" need to trace their way back and e.g. check what you get immediately inside yadism = in the runner e.g. here

yadism/src/yadism/runner.py

Line 127 in 17a6c55

if isinstance(theory["n3lo_cf_variation"], int):

[jekoorn]

on it @felixhekhorn

[Radonirinaunimi]

@Radonirinaunimi I think you put these warnings there in the first place, right? so you know that they are in this region? because this can not be inferred from the output ... (i.e. if we have the kinematics in hand there it might be good to echo them alongside 🙃 )

Yes, I agree. I should have printed out the kinematics alongside for full transparency and certainty. Since we are at it, I would suggest to add this in this PR.

[jekoorn]

However, doing remote debugging is tricky 🙃

I understand, of course, but since it's apparently an issue that only pops up with some FNS's (code runs fine under ZM-VFNS), I was hoping...

But if @evagroenendijk has a runscript with this card that works, so much the better!

[evagroenendijk]

@jekoorn @felixhekhorn Indeed I use exactly that theory card and it works fine. I think it's a compatibility issue between numpy and the scipy integration. I reproduce your error when using numpy v2.4, I myself was using v2.2.6 and then it does work (I think v2.4 numpy outputs an array somewhere while v2.2.6 outputs a value)

[felixhekhorn]

I think it's a compatibility issue between numpy and the scipy integration

ooh, you are saying this is a side effect of our Python (and dependencies) upgrade ... in particular #383

Okay, so we need to sort that out. I would suggest to solve this issue first (starting from master branch) and then rebase/merge this one on top afterwards so the two things get disentangled.

Sigh... if only yadism would have some sort of benchmarks, e.g. here #162 or also just in pinefarm, which would run on a regular basis and where we would detect such problems ...

[jekoorn]

Okay, so we need to sort that out.

Will be addressed in #389

[felixhekhorn]

then rebase/merge this one on top afterwards

please still check explicitly that you get different numbers for different variations (and they are not crazy different)

[jekoorn]

Differences are in the 10-30% range when comparing the correlated vs uncorrelated min/max variations - typically the uncorrelated variations (squares summed) are slightly smaller. I'm thinking about how to properly present the results, but I can just link a file with a plaintext table that I made using pineappl diff.
Btw, all using Eva's nice benchmark runcards.

[felixhekhorn]

Differences are in the 10-30% range when comparing the correlated vs uncorrelated min/max variations

Can we be a bit more specific? for which observable? for which kinematics? we are talking only about the N3LO corrections (and not the full thing), right?

I'm thinking about how to properly present the results, but I can just link a file with a plaintext table that I made using pineappl diff.

If you can come up quickly with some clever representation, fine, else also plain text files will do for now. Then for the 4.1 paper we need to do something more clever 🙃

[jekoorn]

for which observable? for which kinematics?

In this case, I used Eva's runcards for the N3LO benchmark to compute F2C and FLC. To get the N3LO corrections, should I just subtract the NNLO component? And do these observables suffice, or do we want an actual cross-section in the end?

[felixhekhorn]

In this case, I used Eva's runcards for the N3LO benchmark to compute F2C and FLC.

in principle fine, but we need to remember, that some of the kinematics are just unreachable by HERA

And do these observables suffice, or do we want an actual cross-section in the end?

thus, it might make also sense to look at the true settings as well. We can start with the easy thing first (just F2+FL). For the paper we need the real thing.

To get the N3LO corrections, should I just subtract the NNLO component?

Yes. Also PineAPPL can do it directly (I think). Both things (total and relative) are interesting.

[jekoorn]

https://vp.nnpdf.science/rHIQtzZPT32Fi7FTq577zA==/F2C_FLC_all-Q2.pdf

plot here for the F2C and FLC SFs, compared to the NNLO predictions, the shaded band is the old correlated variation, and the colored ones are the new up/down variations for the CFs, both around the same central theory.

[juanrojochacon]

Thanks for the nice plot @jekoorn can you add the IHOU (new) band, to have an idea of how it compares with the IHOU (old) band?

[felixhekhorn]

@jekoorn checked that this agrees with master

[felixhekhorn]

@jekoorn aaah forgot something 🙃 if you are careful you can act directly on master. Please add a Changelog entry. Please add one also for #389 (something like "Fix a bug for evaluation of adani" or similar).

After that (must be done on master): please make a new release. Propagate it to pinefarm. After that can we please recompute all grids such that those have the correct metadata and in particular the correct yadism version? cc @scarlehoff