Make R.swift faster on big projects or string files

[tomlokhorst]

Split off from comment #456 (comment)

R.swift takes 20 seconds to run on a 3000 line strings file. Or 30 seconds to run on a big project.

If anyone has example projects or files they could share, that would be great to reproduce the issue and figure out where the bottleneck is.

/cc @marekpridal @liamnichols

[liamnichols]

@marekpridal: Do you use the .rswiftginore file on your project?

I added some logging into RswiftCore.run to have a quick look at the issue and got the following:

default	15:51:49.168900 +0000	rswift	[T 0.29651403427124023s (total: 0.29651403427124023s)]: Loadeded Xcodeproj
default	15:52:25.139442 +0000	rswift	[T 35.97057890892029s (total: 36.26709294319153s)]: Loadeded IgnoreFile
default	15:52:25.152672 +0000	rswift	[T 0.013236045837402344s (total: 36.28032898902893s)]: Computed resourceURLs
default	15:52:25.202083 +0000	rswift	[T 0.04941093921661377s (total: 36.329739928245544s)]: Initalised Resources
default	15:52:25.202486 +0000	rswift	[T 0.0004290342330932617s (total: 36.33016896247864s)]: Defined generators
default	15:52:25.401039 +0000	rswift	[T 0.19853198528289795s (total: 36.528700947761536s)]: End

In my case, the following line takes 36 seconds to run:

let ignoreFile = (try? IgnoreFile(ignoreFileURL: callInformation.rswiftIgnoreURL)) ?? IgnoreFile()

My ignore file looks like the following:

**/es.lproj/*
**/fa-IR.lproj/*
**/fr.lproj/*
**/hu.lproj/*
**/id.lproj/*
**/pt-BR.lproj/*
**/th.lproj/*
**/vi.lproj/*
**/zh-Hant.lproj/*
**/it.lproj/*
**/el.lproj/*
**/da.lproj/*
**/ru.lproj/*
**/pl.lproj/*
**/hi-IN.lproj/*

Would be good to know if our performance issues are related or not

[marekpridal]

Great work and yes I did! We use it a lot because we use R framework basically only for localizables 👍🏻🙂

[liamnichols]

Looks like we're on the same page then... I've dug into it a bit and it seems like the root cause is in Glob.swift, specifically Glob.expandGlobstar(pattern:) so I guess it's related to using ** in the ignore file.

I don't really know much about all of this but after Googling "glob", I found this doc that says the following:

Note Using the “**” pattern in large directory trees may consume an inordinate amount of time.

I guess it makes sense, our repo is large so there are a lot of directories to check..

Changing my .rswiftignore to the following reduces the overall execute time down to about ~0.5 seconds 🙌

Global/SupportingFiles/es.lproj/*
Global/SupportingFiles/fa-IR.lproj/*
Global/SupportingFiles/fr.lproj/*
Global/SupportingFiles/hu.lproj/*
Global/SupportingFiles/id.lproj/*
Global/SupportingFiles/pt-BR.lproj/*
Global/SupportingFiles/th.lproj/*
Global/SupportingFiles/vi.lproj/*
Global/SupportingFiles/zh-Hant.lproj/*
Global/SupportingFiles/it.lproj/*
Global/SupportingFiles/el.lproj/*
Global/SupportingFiles/da.lproj/*
Global/SupportingFiles/ru.lproj/*
Global/SupportingFiles/pl.lproj/*
Global/SupportingFiles/hi-IN.lproj/*

@marekpridal: Let me know if you are able to do the same and if you experience similar results.

I guess this might not actually be an issue but instead maybe could come under a basic troubleshooting tip or something?

Ps: Running git clean -fd removed a bunch of temporary ignored directories from my local repo (derived data etc) and reduced the ~30 seconds down to ~4 seconds as well but avoiding the globstar completely still gives the best results though

[alexandre-g]

Faced this issue also. With just one line **/CampaignsAndFormats.strings it was increasing script time from 4 to 11 seconds. Maybe should be mentioned in the readme until resolved

https://github.com/mac-cain13/R.swift/blob/master/Documentation/Ignoring.md

[pallzoltan]

To whomever is reading this, I found the --generators parameter very useful. Threw out the wildcarded paths (we had a bunch of those in .rswiftignore) and only use --generators --generators image,string,file,color.
We have 3 targets with R.swift (4 soon), each took 30 seconds before, now its 1s, 5s and 8s.

LE: More about the --generators param: https://github.com/mac-cain13/R.swift/blob/master/Documentation/Ignoring.md
LE2: Just to make it clear - what slowed the process down were the wildcards. Using the generators is just extra performance gain.