Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

onceover directory sync is very slow #215

Open
GeoffWilliams opened this issue Apr 9, 2019 · 0 comments
Open

onceover directory sync is very slow #215

GeoffWilliams opened this issue Apr 9, 2019 · 0 comments

Comments

@GeoffWilliams
Copy link
Contributor

GeoffWilliams commented Apr 9, 2019 •
edited
Loading

Overview

Slow test runs from bundle exec onceover run spec with a long pause before R10K can be seen running

Analysis

Timing the above command consistently gives times > 1minute

real    1m30.620s
user    1m26.584s
sys     0m3.734s

This is due to the .onceover dir sync in lib/onceover/deploy.rb

Working on PR to rewrite this code to be simpler and faster, initial timings on same control repo:

real    0m18.474s
user    0m16.725s
sys     0m1.746s
GeoffWilliams added a commit to GeoffWilliams/onceover that referenced this issue Apr 9, 2019
@GeoffWilliams
(voxpupuli#215) Simplfy and speed up cache building
2afc0ec 
Existing onceover cache building code is slow and this is already noted as a
`TODO` in the code.

Rewrite the internals of the `Onceover::Deploy` class:
1.  Update comments
2.  Build a list of excluded directories using whole paths so they can be
    easily filtered
3.  Build the content for `onceover_manifest.json` as we are processing the
    directory tree, so the `map` operation can be deleted
4.  Completely skip child directories of any directories we exclude (`prune`)
5.  Descend the control repo using Ruby's `Find` module instead of the
    `get_children_recursive` function (deleted)
6.  Process directories and copied files in a single pass
7.  Move the `require` for `json` to live with the other `require` lines
8.  Update the code to dump `onceover-manifest.json`. Since we already exclude
    the root directory we can just write out the whole array as-is

Rest of code left in-place
dylanratcliffe added a commit that referenced this issue Apr 9, 2019
@dylanratcliffe
Merge pull request #216 from GeoffWilliams/cache_speed
620203c 
(#215) Simplfy and speed up cache building
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@GeoffWilliams