Nipype 1 and 2

[satra]

This is more for Nipype 2.0, but i'm sticking it here for now and for 0.1 release. We should push for this once the other 0.1 tasks are done, especially consolidation of the specs API.

• Create a separate project for a nipype 1 to 2 workflow converter.
• Export Nipype 1.0 interfaces to Pydra Task/Node specs

[dPys]

@satra -- Regarding the nipype 1 to 2 workflow converter, is anyone working on this? I'd also be happy to help with that whenever you think pydra is ready for it.

[satra]

@dPys - would love some help. there are a few problems to solve:

1. very short term

one could create a new function nipype1topydratask that would take any Nipype interface and turn it into a pydra task (this has a few things that will need to be thought through, and perhaps at least a few people discussion to iron out a path)

if this function existed, then a nipype workflow could be written out as a pydra flow without too much trouble. all the other semantics map to pydra pretty easily.

related to this is an augment of ShellCommandTask to support better input and output specification. @djarecka and i discussed this, and could be developed after neuroinformatics congress.

another option would be to use pure python workflows (e.g., if you are just working with dipy), and this would already work.

2. medium term

this would involve meta specification support #36, #37 and more generally a nipype 1 interface, nextflo interface/workflow specification translator. some of this exists with nipype2boutiques. i see this as being a place that could make a big difference towards interoperability across these meta languages.

also in this time frame would be some level of resource management, which is currently missing and would be essential for many real world workflows.

3. long term. convert and optimize a nipype1 workflow using pydra's concepts automatically.

@dPys - did you go through the pydra notebooks? if so please post any comments in #147

[dPys]

@satra -- thanks for the updates. It sounds like conversion in pure-python might be the best starting point then? Will work on nipype1topydratask and post in this thread when I have a prototype that I'm happy with.

As for resource management, I'd be willing to work on this aspect over the coming months, particularly since I believe this is where pydra could really shine. On the one hand dask could help a lot, but I'm guessing you'd preferably want pydra to have its own graph scheduler that feeds off of the DAG + runtime statistics provided through Audit?

I have gone through all of the tutorials, and didn't run into any major issues, but will post to #147 if I think of anything new.

FYI: my long-term plan is to revamp PyNets with pydra once a clean internal scheduler is underway.

@dPys

[dPys]

@satra -- please let me know if we are on the right track with nipype1topydratask. @anibalsolon and I put this prototype together at a minihack this afternoon :-)

@dPys

[djarecka]

@dPys @anibalsolon - in case you haven't seen the discussion in #171, we will have a meeting tomorrow at 9am EDT, your feedback would be great if you can join. I will also post notes from the meeting