Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop #10

Merged
merged 28 commits into from
Oct 9, 2018
Merged
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
770f3fc
Add path to save_cache
percyfal Oct 4, 2018
395899c
Update python version
percyfal Oct 4, 2018
4abda0f
Add docker installation
percyfal Oct 4, 2018
b5ae803
Use circleci docker image
percyfal Oct 4, 2018
9369372
Use machine instead of docker in build step
percyfal Oct 4, 2018
4345527
Install python via conda
percyfal Oct 4, 2018
caafa55
Fix typo
percyfal Oct 4, 2018
9b4da9f
Update test environment
percyfal Oct 4, 2018
685b09d
Add base environment to conda update command
percyfal Oct 4, 2018
5ba0322
Add missing dependencies
percyfal Oct 4, 2018
082b732
Add snakemake and use test environment for cache key
percyfal Oct 4, 2018
3bdc040
Remove snakemake version tag
percyfal Oct 4, 2018
c0265f0
Add bioconda channel
percyfal Oct 4, 2018
49ec49b
Modify tests to comply with docker-py >= 3.0
percyfal Oct 4, 2018
4bf499e
Remove working directory
percyfal Oct 5, 2018
149ae45
Restore cache prior to miniconda installation
percyfal Oct 5, 2018
ee1167c
Add slurm-submit-advanced.py to deal with advanced submissions
percyfal Oct 9, 2018
b0dc612
Add cookiecutter option and test for slurm submission script
percyfal Oct 9, 2018
79666c5
Make advance submission script executable
percyfal Oct 9, 2018
f02d953
Update slurm test for new setup
percyfal Oct 9, 2018
2631783
Add tests for advanced submission script
percyfal Oct 9, 2018
f0fe227
Add more tests for slurm advanced
percyfal Oct 9, 2018
2cb9995
Add support for mem_mb in slurm-submit
percyfal Oct 9, 2018
6065762
Add cluster configuration file for tests
percyfal Oct 9, 2018
5983447
Update readme
percyfal Oct 9, 2018
7ac92b6
Delete travis config and migrate to circleci
percyfal Oct 9, 2018
ae1412d
Merge branch 'feature/check-resources-at-submission' into develop
percyfal Oct 9, 2018
80c797b
Add circleci badge
percyfal Oct 9, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update readme
  • Loading branch information
percyfal committed Oct 9, 2018
commit 598344741e7a68b6256e2f5a2252462b1a1cb86b
46 changes: 45 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,46 @@ Then, you can run Snakemake with
snakemake --profile slurm ...


### Submission scripts

There are two submission scripts that configure jobs for job
submission. `slurm-submit.py`, the default, will use the supplied
parameters untouched, making no attempt to modify the job submission
to comply with the cluster partition configuration. On the other hand,
`slurm-submit-advanced.py` will attempt to modify memory, time and
number of tasks if these exceed the limits specified by the slurm
configuration. If possible, the number of tasks will be increased if
the requested memory is larger than the available memory, defined as
the requested number of tasks times memory per cpu unit.

### Resources

The following resources are supported by on a per-rule basis:

- **mem**: set the memory resource request in mb.
- **mem**, **mem_mb**: set the memory resource request in mb.
- **walltime**, **runtime**: set the time resource in min.

### Cluster configuration file

As of 2018-10-08, the profile supports setting options in the cluster
configuration file. The options must be named according to [sbatch
long option names](https://slurm.schedmd.com/sbatch.html). Note that
setting memory or time will override resources specified by rules.
Therefore, setting time or memory in the `__default__` section will
effectively override all resource specifications that set these
parameters.

As an example, you can specify constraints in the cluster
configuration file:

```yaml
__default__:
constraint: mem500MB

large_memory_requirement_job:
constraint: mem2000MB
```


## Tests

Expand All @@ -41,3 +74,14 @@ Cookiecutters](https://github.com/hackebrot/pytest-cookies). You can
run the tests by issuing

pytest -v -s


## ChangeLog

### 2018-10-09

- add support for mem_mb in resources
- add support for cluster configuration file
- add advanced slurm-submit file
- adjust resource requirements if they exceed partition configuration
settings (#11)