dategen in three phases with slurm job

[hyjorc1]

1. Download repositories: GitHubRepoBareDownloader.java

2. Generate seq files for each project: SeqRepoGenerator.java, SeqRepoBuilder.java, run-generator.sh, slurmJob.sh

3. Combine seq files: SeqRepoCombiner.java, run-combiner.sh

[hridesh]

What would be the impact of this change on local data generation, e.g. on just a single machine owned by Boa user?

[hridesh]

If Boa code is intended to compile using Java 7 or higher, one could use concurrency features such as ExecutorService that is meant to provide all of these features. Just a thought. See https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html

[hridesh]

This could be merged provided it works for the local data-generation. The enhancement to use advanced concurrency features could be done later.

[hridesh]

If Boa code is intended to compile using Java 7 or higher, one could use concurrency features such as ExecutorService that is meant to provide all of these features. Just a thought. See https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html

[hyjorc1]

These changes fix and optimize the JSON retriever for extracting repo metadata.

GetReposByLanguage.java takes 4 arguments:

1. The path to the token file.
2. JSON files (100 repositories per file) output path.
3. A number to which the repository stars are greater than or equal to.
4. The programming languages (split by semicolons, e.g. "java;python") used by the repository.