eko 0.15 breaks compatibility with certain grids

[scarlehoff]

After merging #205 certain fktables for the 4.1 theory cannot be generated. The cause still under investigation but opening the issue as a heads up.

pineko theory opcards 41000000 ATLAS_Z0J_8TEV_PT-Y --overwrite                                                                                         
zAnalyze ATLAS_Z0J_8TEV_PT-Y                                                                                                                                                                                                                
                                                                                                                                                                                                                                            
thread '<unnamed>' panicked at pineappl/src/subgrid.rs:216:42:                                                                                                                                                                              
slice index starts at 7 but ends at 0                                                                                                                                                                                                       
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace                                                                                                                                                               
Traceback (most recent call last):                                                                                                                                                                                                          
  File "/scratch/jcruzmar/nnpdf_theory_folder/nnpdftheoryvortex/bin/pineko", line 8, in <module>                                                                                                                                            
    sys.exit(command())                                                                                                                                                                                                                     
  File "/scratch/jcruzmar/nnpdf_theory_folder/nnpdftheoryvortex/lib64/python3.9/site-packages/rich_click/rich_command.py", line 378, in __call__                                                                                            
    return super().__call__(*args, **kwargs)                                                                                                                                                                                                
  File "/scratch/jcruzmar/nnpdf_theory_folder/nnpdftheoryvortex/lib64/python3.9/site-packages/click/core.py", line 1157, in __call__                                                                                                        
    return self.main(*args, **kwargs)                                                                                                                                                                                                       
  File "/scratch/jcruzmar/nnpdf_theory_folder/nnpdftheoryvortex/lib64/python3.9/site-packages/rich_click/rich_command.py", line 166, in main                                                                                                
    rv = self.invoke(ctx)                                                                                                                                                                                                                   
  File "/scratch/jcruzmar/nnpdf_theory_folder/nnpdftheoryvortex/lib64/python3.9/site-packages/click/core.py", line 1688, in invoke                                                                                                          
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                 
  File "/scratch/jcruzmar/nnpdf_theory_folder/nnpdftheoryvortex/lib64/python3.9/site-packages/click/core.py", line 1688, in invoke                                                                                                          
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                 
  File "/scratch/jcruzmar/nnpdf_theory_folder/nnpdftheoryvortex/lib64/python3.9/site-packages/click/core.py", line 1434, in invoke                                                                                                          
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                          
  File "/scratch/jcruzmar/nnpdf_theory_folder/nnpdftheoryvortex/lib64/python3.9/site-packages/click/core.py", line 783, in invoke                                                                                                           
    return __callback(*args, **kwargs)
  File "/scratch/jcruzmar/repositories/pineko/src/pineko/cli/theory_.py", line 34, in opcards
    theory.TheoryBuilder(theory_id, datasets, overwrite=overwrite).opcards()
  File "/scratch/jcruzmar/repositories/pineko/src/pineko/theory.py", line 286, in opcards
    self.iterate(self.opcard, tcard=tcard)
  File "/scratch/jcruzmar/repositories/pineko/src/pineko/theory.py", line 254, in iterate
    f(name, grid, **kwargs)
  File "/scratch/jcruzmar/repositories/pineko/src/pineko/theory.py", line 274, in opcard
    _x_grid, q2_grid = evolve.write_operator_card_from_file(
  File "/scratch/jcruzmar/repositories/pineko/src/pineko/evolve.py", line 108, in write_operator_card_from_file
    pineappl_grid.optimize()
pyo3_runtime.PanicException: slice index starts at 7 but ends at 0

[Radonirinaunimi]

I had seen issues similar to this before but I don't seem to find a trace of it in the recent issues. This basically happens because one optimizes a grid with empty subgrid(s). Is there something particular with that grid (that prevents one from optimizing it)? As usual, it'd be greatly helpful if you add a link to get the grid.

[scarlehoff]

The grids are those in theory slim as usual (and you already have them, we have already iterated over these grids xD)

This error, for this same grids, was already seen and fixed: NNPDF/pineappl#338 (comment)

so I'm not sure why it came back. Perhaps it was fixed for fktables and not grids?

Note that it not only breaks these grids but also some of the old ones (that we might not care about anymore, but also have empty subgrids)

[Radonirinaunimi]

This error, for this same grids, was already seen and fixed: NNPDF/pineappl#338 (comment)

so I'm not sure why it came back. Perhaps it was fixed for fktables and not grids?

NNPDF/pineappl#338 fixed the issue of empty subgrids by clearing the array right after the scaling (in this case, scaling by zero for some of the subgrids). Clearing the array was crucial in order for this check to be triggered (for the technical details, see NNPDF/pineappl#338 (comment)).

But, somehow, empty subgrids have been introduced in the grids without the array being cleared. Were there further operations that have been done to the grids?

@cschwan, I was wondering whether, to be fully bulletproof, we should also check that

subgrid.indexed_iter().collect::<Vec<_>>()

is not empty (in addition to subgrid.is_empty()?

[cschwan]

@cschwan, I was wondering whether, to be fully bulletproof, we should also check that

subgrid.indexed_iter().collect::<Vec<_>>()

is not empty (in addition to subgrid.is_empty()?

Can you point me to where exactly we should check that?

[scarlehoff]

These are the same grids as back then.

So, if I understand correctly, the fktables worked because they were created from the beginning with the new version, but the old grids, which have 0 subgrids, won't work. Is that correct?

If so we can simply wrap the optimize in a try-error and "skip over" this problem, as we are interested in optimizing fktables, grisa we can do without.

[Radonirinaunimi]

Can you point me to where exactly we should check that?

I was thinking inside optimize_subgrid_type where check for subgrid.indexed_iter().collect::<Vec<_>>().len() == 0 instead/in addition of subgrid.is_empty().

So, if I understand correctly, the fktables worked because they were created from the beginning with the new version, but the old grids, which have 0 subgrids, won't work. Is that correct?

I am not fully sure anymore... There might have been something introduced between PineAPPL v1.0.0 and v1.1.0 that triggered similar/part of this problem. The benchmark now fails in #227 (with old and old Integrability grid) while it works on main.

[cschwan]

Sorry I saw this too late, but this is a panic in PineAPPL and rhus automatically a bug (in PineAPPL). I'll have a look at it this evening!

[cschwan]

I was thinking inside optimize_subgrid_type where check for subgrid.indexed_iter().collect::<Vec<_>>().len() == 0 instead/in addition of subgrid.is_empty().

I think we should solve this a bit differently, because subgrid.is_empty() should imply subgrid.indexed_iter().collect::<Vec<_>>().len() == 0. The way I would fix this problem is as follows:

1. Promote the condition above as an invariant, and check it with an assertion statement in the code before the line where the reported panic happens. Also document in the assertion message how to fix the problem (see next point).
2. To fix the problem, I'd write a new method bool Grid::repair() and add the CLI switch pineappl write --repair ..., which repair the affected subgrids. The method would return true if it actually repaired something or false if the grid is OK.
3. To put this into use, we can
   1. repair the affected grids and add them to/replace them in the theories, or
   2. we simply let Pineko call repair before optimize if you decide to keep optimize.

Having Grid::repair has the advantage that we can easily add more operations that repair 'broken' grids, should something like this happen again. What do you think?

[Radonirinaunimi]

Hi @cschwan, yes, I do like very much your proposition, especially the part with Grid::repair.

[felixhekhorn]

@cschwan did you have by chance already have a look? is there something I could help with?

The above comment doesn't really give an explanation of what is happening. Also, why do you think it is a grid (file object) problem instead of an programming (API) problem? And if I may pick up the role of the devil's advocate for a moment: does a Grid::repair imply that PineAPPL is not 100% backward compatible (as claimed)?