Multiprocess access to the same file or folder

[hosseinfani]

@Hamedloghmani
When running Adila in parallel for multiple *.pred files, we rely on file locking system of os. However, there will be some issues like

One process may be true in condition check, but when creating see the folder exists due to other process!

Adila/src/main.py

Line 170 in 040b34b

if not os.path.isdir(output): os.makedirs(output)

Or when logging the time at:

Adila/src/main.py

Line 197 in 040b34b

with open(f'{output}/rerank.time', 'a') as file: file.write(f'{elapsed_time} {new_output} {algorithm} {k_max}\n')

Temporary solution is to run sequentially for one pred file. Then, run in parallel for all files.

[Hamedloghmani]

@hosseinfani
Thank you for pointing that out. That was a really solid point. I think this was the reason that on unix processes became zombies and running would not finish in some cases, although Windows handled that.
I will look for a permanent solution.

[hosseinfani]

@Hamedloghmani
put it in the lowest priority, pls. Not important at all for now