Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry during heavy load results in Retry Hell #15

Open
TheCrabity opened this issue Feb 4, 2018 · 3 comments
Open

Retry during heavy load results in Retry Hell #15

TheCrabity opened this issue Feb 4, 2018 · 3 comments
Labels

Comments

@TheCrabity
Copy link

TheCrabity commented Feb 4, 2018

When a Worker is hitting the Rate Limit, we don't respect maxTries and the worker can enter a Retry Hell, with 30 or more seconds before the promise is returned. Have an option to throw an error (and handle retries ourself), or retry only as many times as maxTries would prevent this.

https://github.com/pogosandbox/pogobuf/blob/master/pogobuf/pogobuf.client.js#L610

Temp work around that corrects this issues:

            if (responseEnvelope.status_code === 52 && self.endpoint !== INITIAL_ENDPOINT) {
                  throw new retry.StopError(
                    `Status code ${responseEnvelope.status_code} received from RPC`
                );
            }
@niicojs
Copy link
Collaborator

niicojs commented Feb 11, 2018

Could you try https://github.com/pogosandbox/pogobuf/tree/throttlinghell ?
Just a attempt, I didn't test much

@niicojs niicojs added the bug label Feb 11, 2018
@TheCrabity
Copy link
Author

image

Setting maxTriesThrottling: to 0 or 1, still results in the odd double retry attempt, as seen above.

@niicojs
Copy link
Collaborator

niicojs commented Feb 18, 2018

I don't really get how that's possible.
Could you debug this part or add log to see what's going on?

if (responseEnvelope.status_code === 52 && self.endpoint !== INITIAL_ENDPOINT) {
    self.throttled++;
    if (self.throttled < self.options.maxTriesThrottling) {
        return Promise.delay(2000).then(() => self.tryCallRPC(requests, signedEnvelope));
    } else {
        throw new retry.StopError(`Throttled ${self.throttled} times`);
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants