Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The job stops restarting workers and exits if the traceback is a code bug. #1068

Open
workingloong opened this issue Apr 8, 2024 · 4 comments
Open

The job stops restarting workers and exits if the traceback is a code bug. #1068

workingloong opened this issue Apr 8, 2024 · 4 comments
Assignees
@BalaBalaYi
Labels
enhancement New feature or request question Further information is requested todo issue or pr with 'todo' will ignore expiration
Milestone
Backlog

Comments

@workingloong
Copy link
Collaborator

The restarted worker will fail again if the training fails due to a code bug. The job should exit as soon as possible to release resources on a cluster.
@BalaBalaYi BalaBalaYi self-assigned this Apr 22, 2024
@BalaBalaYi BalaBalaYi added bug Something isn't working question Further information is requested labels Apr 22, 2024
@BalaBalaYi
Copy link
Collaborator

A blacklist mechanism can be introduced to this case: throw an explicit error for user code errors with no more retry and free up resources.

@BalaBalaYi
Copy link
Collaborator

BalaBalaYi commented Apr 24, 2024 •
edited
Loading

user_code_bug_demo_log.txt

@BalaBalaYi BalaBalaYi added enhancement New feature or request and removed bug Something isn't working labels Apr 24, 2024
@BalaBalaYi BalaBalaYi added this to the Backlog milestone Jun 6, 2024
@github-actions GitHub Actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity.

@github-actions github-actions bot added the stale label Oct 14, 2024
@github-actions GitHub Actions
Copy link

This issue is being automatically closed due to inactivity.

@BalaBalaYi BalaBalaYi added todo issue or pr with 'todo' will ignore expiration and removed stale labels Nov 18, 2024
@BalaBalaYi BalaBalaYi reopened this Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BalaBalaYi BalaBalaYi

Labels
enhancement New feature or request question Further information is requested todo issue or pr with 'todo' will ignore expiration
Projects
None yet
Milestone
Backlog
Development

When branches are created from issues, their pull requests are automatically linked.

1068-the-job-stops-restarting-workers-and-exits-if-the-traceback-is-a-code-bug intelligent-machine-learning/dlrover
2 participants
@workingloong @BalaBalaYi