NXlauetof format for NMX data

[aaronfinke]

Executive summary

NMX Data output should be in the nexus-nxmx format

Context and background knowledge

In order to comply with modern data handling and archiving standards for macromolecular crystallography (MX) data, the data output should be in the nexus nxmx format (https://manual.nexusformat.org/classes/applications/NXmx.html). This format is already used at most synchrotron X-ray MX beamlines around the world, and NMX should comply with these standards as much as is possible. The nxmx format is generic to source and is inclusive of neutron MX data. This format is part of the MX "Gold standard" for data, adopted in 2020 after a long deliberation process. https://journals.iucr.org/m/issues/2020/05/00/ti5018/index.html

Inputs

All data properties are detailed here: https://manual.nexusformat.org/classes/applications/NXmx.html

Note that not all fields are required, and is experiment-specific. Others that are labeled "recommended" are absolutely necessary for NMX data processing. Determining which are required for NMX data is ongoing.

Methodology

For the most part, this will require accessing data from e.g. EPICS PVs and parsing them and putting them in the correct format. Some calculations might be required for determination of orientation- this is in progress.

Outputs

Data arranged in the correct format (see above).

Which interfaces are required?

Integrated into reduction workflow, Python module / function

Test cases

We have simulation data from McStas that we can use for testing.

Comments

No response

[aaronfinke]

After discussion, we will change the requested format to the NXLauetof format: https://manual.nexusformat.org/classes/applications/NXlauetof.html

[SimonHeybrock]

That application definition contains required fields such as polar_angle which are kind of deprecated in NeXus. ESS files use NXtransformations. @aaronfinke can you clarify what to do about this?

[aaronfinke]

Polar angle and azimuthal angle for each detector (the center of the detector) can and should be calculated relative to the goniometer position which will have an NXtransformations entry. If it's a spherical coordinate system then the polar angle is theta and the azimuthal angle is phi. I think DIALS is expecting these values so they should be populated.

[SimonHeybrock]

If it's a spherical coordinate system then the polar angle is theta and the azimuthal angle is phi.

Note to implementer: Thistheta is not the theta from Bragg's law, but twice that.

[SimonHeybrock]

Based on discussion:

• Write NXlauetof with three NXdetector groups.
• Run workflow in a loop, write result groups one by one to avoid using too much memory from handling all files at once.
• Skip control group for now.
• Existing file writer can be replaced/removed.

[jokasimr]

Based on discussion:
Write NXlauetof with three NXdetector groups.
Run workflow in a loop, write result groups one by one to avoid using too much memory from handling all files at once.
Skip control group for now.
Existing file writer can be replaced/removed.

Some questions about this:

Write NXlauetof with three NXdetector groups.

1. Did I understand this right that we should have one NXdetector per 2D detector of the NMX instrument?

Run workflow in a loop, write result groups one by one to avoid using too much memory from handling all files at once.

2. Is the loop over files to be reduced or is the loop over the planar 2D detectors?
3. Is each "result group" an NXdetector entry or something else?

[aaronfinke]

1. Did I understand this right that we should have one NXdetector per 2D detector of the NMX instrument?

Correct.

2. Is the loop over files to be reduced or is the loop over the planar 2D detectors?

Loop over the planar 2D detectors.

3. Is each "result group" an NXdetector entry or something else?

Yes, each result group is an NXdetector entry.

[YooSunYoung]

@jokasimr did you pick this up? I forgot to open smaller issues and assign myself to it....
Maybe we can split it into two parts, 1. applying NXlauetof format and 2. looping over 2d detectors and save it into different detector groups.

[jokasimr]

No I didn't pick it up yet 👍

[YooSunYoung]

I'll split this into 3 issues.

• NXlauetof format for NMX data - export function update, fixing names and dtypes, and add placeholders. #96
• Make sure all fields are calculated as expected including unit tests and integration test.
• Update workflow object and example to loop over each panel for limiting memory consumption.

[SimonHeybrock]

I additionally suggest a preparatory step, removing the current map operation over panels from the workflow. It will make the subsequent changes simpler and easier to review.