Nipype1 interface wrapper and simple workflow

[effigies]

Paths may be an issue...

[effigies]

Some initial fiddling:

import pydra
import nipype
import dataclasses as dc
import typing as ty
from nipype.interfaces import fsl

def traitedspec_to_specinfo(traitedspec):
    return pydra.specs.SpecInfo(
        name="Inputs",
        fields=[
            (name, ty.Any, dc.field(metadata={"help_string": trait.desc}))
            for name, trait in traitedspec.traits().items()
        ]
    )

I don't really see how to create a task except by subclassing TaskBase or @pydra.mark.task, so I'm going with subclassing.

class Nipype1Task(pydra.TaskBase):
    """Wrap a Python callable as a task element."""

    def __init__(
        self,
        interface: nipype.interfaces.base.BaseInterface,
        audit_flags: pydra.AuditFlag = pydra.AuditFlag.NONE,
        cache_dir=None,
        cache_locations=None,
        messenger_args=None,
        messengers=None,
        name=None,
        **kwargs,
    ):
        self.input_spec = traitedspec_to_specinfo(interface.input_spec())
        self._interface = interface
        if name is None:
            name = interface.__class__.__name__
        super(Nipype1Task, self).__init__(
            name,
            inputs=kwargs,
            audit_flags=audit_flags,
            messengers=messengers,
            messenger_args=messenger_args,
            cache_dir=cache_dir,
            cache_locations=cache_locations,
        )
        self.output_spec = traitedspec_to_specinfo(interface.output_spec())

    def _run_task(self):
        inputs = dc.asdict(self.inputs)
        res = self._interface.run(**inputs)
        self.output = res.outputs

This isn't quite ideal, because it means that Nipype1Task(fsl.BET()) is a one-use thing, when the goal would be something like

newBET = Nipype1Task(fsl.BET)
bet = newBET(...)

Anyway, I'm getting stuck on actually running the thing:

wf = pydra.Workflow(name='unifize_and_skullstrip_wf', input_spec=['in_file'])
bet = Nipype1Task(fsl.BET())
bet.in_file = wf.lzin.in_file
wf.add(bet)

with pydra.Submitter() as sub:
    sub(wf)

The results in:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-22-a8fc839fa509> in <module>
      1 with pydra.Submitter() as sub:
----> 2     sub(wf)

~/Projects/nipype2/pydra/pydra/engine/submitter.py in __call__(self, runnable, cache_locations)
     48                 if nd.allow_cache_override:
     49                     nd.cache_dir = runnable.cache_dir
---> 50             runnable.inputs._graph_checksums = [
     51                 nd.checksum for nd in runnable.graph_sorted
     52             ]

~/Projects/nipype2/pydra/pydra/engine/submitter.py in <listcomp>(.0)
     49                     nd.cache_dir = runnable.cache_dir
     50             runnable.inputs._graph_checksums = [
---> 51                 nd.checksum for nd in runnable.graph_sorted
     52             ]
     53         if is_workflow(runnable) and runnable.state is None:

~/Projects/nipype2/pydra/pydra/engine/core.py in __getattr__(self, name)
    187         if name == "lzout":  # lazy output
    188             return LazyField(self, "output")
--> 189         return self.__getattribute__(name)
    190 
    191     def help(self, returnhelp=False):

~/Projects/nipype2/pydra/pydra/engine/core.py in checksum(self)
    207             self.inputs._graph_checksums = [nd.checksum for nd in self.graph_sorted]
    208 
--> 209         input_hash = self.inputs.hash
    210         if self.state is None:
    211             self._checksum = create_checksum(self.__class__.__name__, input_hash)

AttributeError: 'Inputs' object has no attribute 'hash'

[satra]

@effigies - before we go down this rabbit hole, we should try to determine if we are going to move forward with attr #174 or not. i am hoping @djarecka can fix the issues that are ailing the 3 failing tests.

[effigies]

When people get a minute, maybe we can discuss the status of this, post attr.ib migration?

[satra]

    my_input_spec = SpecInfo(
        name="Input",
        fields=[
            (
                "text",
                attr.ib(
                    type=str,
                    metadata={"position": 1, "help_string": "text", "mandatory": True},
                ),
            )
        ],
    )

[dafrose]

How is the current progress on this issue? I suggested the introduction of a yaml-interface to create tasks for command line tools (primarily, not exclusively) here #367 . A possible route to go from nipype1 to pydra could be this:

1. Translate nipype1 TraitedSpecs to yaml files.
2. Parse yaml files to create tasks in pydra.

Advantages:

• Enables users to also translate their own custom-written nipype1 interfaces automatically.
• Resulting yaml files can be kept for the future, reducing processing time when reusing the interface.
• Resulting collection of yaml files can be easily packaged as pydra addon-packages, solving pydra interface packages from nipype 1 #170 and making it easy to completely detach from original nipype1 code.
• The compatibility of the resulting yaml files with pydra can be tested at both ends: when serializing into yaml and when parsing into tasks, making it easy to see, where something went wrong.

Disadvantages:

• Conversion creates new files, so file saving permission is needed and some path must be specified. However, this could be mitigated by allowing a user to directly feed the yaml stream to the parser in pydra instead of saving the files.

Since workflows are tasks in pydra, it would make sense to do the same for workflows. Regarding tests, one could rewrite original nipype1 tests to use yaml-based pydra tasks instead.

What do you think?

[dafrose]

I just noticed that you already implemented a nipype1 task wrapper with the PR nipype/pydra-nipype1#1
Has this been tested?

[effigies]

@dafrose Yes, a method to translate the nipype1 packages to YAML files that could be loaded as Pydra tasks would be fantastic. The Nipype1Task does work, though I'm not sure where I have an example workflow (this should be added in the CI), but the disadvantage is that the build-time type checking is completely missing, and all trait checks will be performed at runtime.

Converting the input and output specs automatically via a YAML representation seems useful, as it would give us a declarative, Python-independent representation. The only problem I see is with interfaces that have more complicated internal workings that cannot be determined only from input and output specs.

[dafrose]

@effigies happy to hear that you like the idea :-)

The only problem I see is with interfaces that have more complicated internal workings that cannot be determined only from input and output specs.

Do you have something in particular in mind? Anything that strictly requires user input via the command line during runtime does not work with nipype1 anyway, does it? Regarding more complex relations between inputs and output filenames, it would always be possible to reference any Python function from the YAML spec. Anything beyond that?

[effigies]

Here's an example of something that would be painful: https://github.com/nipy/nipype/blob/07af08f98/nipype/interfaces/freesurfer/preprocess.py#L1010-L1569

ANTs tools are often other examples. They may be doable within the YAML, but I think the YAML would need to be designed, not automatically generated from what currently exists.

[djarecka]

i just want to point you to my branch where I'm working on fsl package. My way of doing is to have a converter that needs nipype1 interface and a hand-written yml file for things that I can't automatically take from nipype.
The converted interfaces are here. I still have to add some additional fields to my yml file that I don't now how to cover automatically.

@effigies @dafrose - perhaps we could zoomeet one day, what do you think?

[effigies]

Can generally call in the 9-11am (Eastern US) range. In the next two weeks, any days but Friday Oct 30 and Tuesday Nov 3 work.

If you need more precision, can you send a poll?