Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deliver end-of-subworkflow file cleanup for files that are not output #5222

Open
unito-bot opened this issue Feb 28, 2025 · 2 comments
Open
Assignees

Comments

@unito-bot
Copy link

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1702

@adamnovak
Copy link
Member

This could get most of the benefits of #4872, since if you have a file you won't need later in your workflow you can split out a sub-workflow to scope it and the file will go away when the sub-workflow ends. And it doesn't need any reference-counting or hooking into expression evaluation or weird un-designed synchronization primitives between jobs, just the ability to track all files created (and gather them at the end of the workflow), and the ability to dig through the output values, and the ability to delete what's in one set and not the other.

@adamnovak
Copy link
Member

If we keep the current environment-only-grows system, the environment could be the all-files-ever-created collection, and we wouldn't need a separate one. I liked the simplification I was working on in https://github.com/adamnovak/toil/tree/issues/4872-drop-wdl-files, where a job will produce an environment with just what it actually defines itself, but maybe that's not needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants