Raygent simplifies parallel execution in Python by providing an intuitive interface to Ray's distributed computing framework. It removes boilerplate code and offers a modular approach to managing parallel tasks, making it easier to scale your computations across multiple cores or nodes.
- Simple Task-Based API: Define your computational logic once and automatically scale across cores.
- Flexible Execution Modes: Run tasks either in parallel (using Ray) or sequentially with a single parameter.
- Resource Optimization: Automatically detect available CPU cores and manage resource allocation.
- Chunked Processing: Efficiently process data in optimal batch sizes.
- Intermediate Result Saving: Save results at customizable intervals with pluggable savers.
- Error Handling: Built-in error capture and logging.
Parallelizing code with Ray requires significant boilerplate and infrastructure management. Raygent abstracts away this complexity with a clean, task-focused API that lets you concentrate on your actual computation logic rather than parallelization mechanics.
from raygent import Task, TaskManager
# Define your task
class SquareTask(Task):
def process_item(self, item):
return item ** 2
# Create a task manager
manager = TaskManager(SquareTask, use_ray=True)
# Process items in parallel
manager.submit_tasks(items=[1, 2, 3, 4, 5])
results = manager.get_results()
print(results) # [1, 4, 9, 16, 25]Clone the repository:
git clone [email protected]:oasci/raygent.gitInstall raygent using pip after moving into the directory.
pip install .This will install all dependencies and raygent into your current Python environment.
We use pixi to manage Python environments and simplify the developer workflow.
Once you have pixi installed, move into raygent directory (e.g., cd raygent) and install the environment using the command
pixi installNow you can activate the new virtual environment using
pixi shellThis project is released under the Apache-2.0 License as specified in LICENSE.md.