Make task file compilation more intelligent.

[cowboy]

This might be a duplicate.

This is how I see it working. When Grunt runs a task:

1. Grunt gives the task an array of known src/dest pairs and just src files (what it already does with this.files and this.filesSrc).
2. The task gives Grunt back an object that tells Grunt about additional dependencies any src file might have. (:star:)
3. Grunt checks their (and their dependencies') mtimes in its index-of-files-to-mtimes, and updates the this.files and this.filesSrc to remove files that don't need to be compiled due to mtimes.
4. The task does its thing, keeping track of what files were successfully compiled/linted/whatever.
5. The task gives Grunt an array of successfully compiled/linted/whatever src files.
6. Grunt updates the mtimes in its index-of-files-to-mtimes.

Steps 2-3 could be skipped by plugins that won't have additional dependencies. Also, for plugins that might create their list of dependencies as-they-go, and not up-front, we could possibly expose the mtime-checking method behind step 3 to the task.

Steps 4-6 could be skipped by plugins that don't compile/lint or don't operate on src files.

⭐ Assuming Grunt passes in src files src/simple.css and src/has-imports.css:

// This object could look something like:
{
  "src/simple.css": ["src/simple.css"],
  "src/has-imports.css": ["src/has-imports.css", "src/import1.css", "src/import2.css"]
}

// Although maybe specifying the "key" src filename in the
// array is redundant. Maybe it could be more like:
{
  "src/simple.css": [],
  "src/has-imports.css": ["src/import1.css", "src/import2.css"]
}

// Or even just this, and Grunt can assume it has to check
// src/simple.css since it was in the original src list:
{
  "src/has-imports.css": ["src/import1.css", "src/import2.css"]
}

[cowboy]

FWIW, I'm working with @shama on creating a simple mtime-checking lib that would either work on an in-memory or filesystem-based index-of-files-to-mtimes. It would:

• Be able to filter a list of src files, removing any whose mtime isn't newer.
• Be able to update the index-of-files-to-mtimes to reflect the mtime of files that have been processed.

Possible options:

• mtime vs shasum vs custom function that accepts 2 filepaths (or something similar)
• fs vs object in memory for persisting mtime/shasum/etc info

[ryanflorence]

related implementation description from @wycats https://gist.github.com/wycats/ed9cd0d0f7ce07537e70

[shama]

Something related we should keep on eye on is this idea: https://gist.github.com/dominictarr/5990143 For quickly comparing large file trees diffs.

[cowboy]

FWIW, @wycats suggested to me in IRC that my proposal here is tied to the grunt-pipeline proposal that @rpflorence linked. I'm not sure how tied-together they need to be; I think they can be 2 separate things.

I understand the idea of "pipelines" vis-a-vis Grunt to be a way of simplifying task configuration so that interim file creation between tasks can be automated.

Basically, I see 2 things that need to happen to make everyone happy:

1. Making builds faster via smart compilation based on an arbitrary "has x src file changed since last compile?" comparison, which I have tried to address in the above proposal.
2. Making builds simpler to configure whereby interim tempfiles that need to be created between a series of tasks can be done automatically, which has yet to be proposed.

@wycats @rpflorence, if my assumptions are incorrect, please correct them. I'd really like solve the "making builds faster" problem first (in a way that pipelines can use but that is not specific to pipelines), followed by the "making builds simpler to configure" problem, which I'm aware of but is a far less requested feature.

[cowboy]

@shama, I know you started working on a mtime-checking lib... have you made any progress on it?

[hurrymaplelad]

Watchify's interaction with 2 seems interesting.

Watchify speeds up rebuilds on file changes by keeping the module dependency tree in memory. Building that tree from source files is expensive.

We could speed up watchify / browserify startup time by writing the dependency tree to a file, but we'd need to check if any of the files has changed before we can use the file-cached dependencies.

What do folks think of Grunt giving tasks the last reported dependency tree in step 1 if it has one, flagging any files that have been modified?

[doberkofler]

+1

[tomByrer]

Still in que for 0.5?

[hurrymaplelad]

For a short-term fix, I've been using grunt-find + grunt-ttime to quickly filter a task's files to those modified since the last time the task completed.

Depends on the shell find command, and configuration is a little hairy, but it's fast.