feat(RHOAIENG-26480): Run RayJobs against existing RayClusters

[chipspeak]

Issue link

Jira

What changes have been made

Support added to the CodeFlare SDK to leverage the upstream KubeRay RayJob Python Client

Verification steps

1. Log into OC via the CLI on your local machine (required as python client needs a kubeconfig)
2. Build a whl file for the CodeFlare SDK based on this branch.
3. Create a Jupyter Notebook locally and paste the below code into cells as labelled:

Cell 1

# This obviously presumes the location of your whl file. Adjust as needed.
%pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ dist/codeflare_sdk-0.0.0.dev0-py3-none-any.whl --force-reinstall

NOTE: Ensure that you restart the notebook kernel after installing the whl file.

Cell 2

from codeflare_sdk import Cluster, ClusterConfiguration

cluster = Cluster(ClusterConfiguration(
    name='test-cluster',
    namespace='rhods-notebooks',
    head_cpu_requests='1',
    head_cpu_limits='2',
    head_memory_requests=4,
    head_memory_limits=5,
    head_extended_resource_requests={'nvidia.com/gpu':0},
    worker_extended_resource_requests={'nvidia.com/gpu':0},
    num_workers=1,
    worker_cpu_requests='1',
    worker_cpu_limits='2',
    worker_memory_requests=3,
    worker_memory_limits=4,
))

Cell 3

from codeflare_sdk.ray import RayJob

# Create and submit a Ray job
rayjob = RayJob(
    job_name="sdk-test-job",
    cluster_name="test-cluster",
    namespace="rhods-notebooks",
    entrypoint="python -c 'import time; time.sleep(20)'",
)

rayjob.submit()

4. Execute the above python code (ensuring the RayCluster is up before executing cell 3).
5. The RayJob should have executed successfully when you check your cluster.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.64%. Comparing base (0ac44ed) to head (7abeca4).
⚠️ Report is 2 commits behind head on ray-jobs-feature.

Additional details and impacted files

@@                 Coverage Diff                  @@
##           ray-jobs-feature     #867      +/-   ##
====================================================
+ Coverage             92.47%   92.64%   +0.16%     
====================================================
  Files                    24       26       +2     
  Lines                  1396     1427      +31     
====================================================
+ Hits                   1291     1322      +31     
  Misses                  105      105              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kryanbeane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kryanbeane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[laurafitzgerald]

These changes look good! I've reviewed the code and some comments inline.

A note on the verification. One comment on the output of the job submission, i think the red output could be read as an error but seems like all ran fine. The job completed fine even though I did attempt to submit the job before the cluster worker pos were ready.

[laurafitzgerald]

this log probably needs an update or to be moved. the RayJob doesn't get initialized it get's submitted/applied.

[chipspeak]

Yeah I just have that for initial clarity as we've initialised the object but not submitted it. I can probably remove the logger stuff as is really for now until we decide on an output format?

[laurafitzgerald]

i think this one is being removed elsewhere?

[chipspeak]

It's added in the new RayJob logic so it has to be here for the unit test assertions I think!