Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Support Discord async and fix its ratelimiting, improve flexibility o…
…f Orca's core functionalities, and more (#753) * style(examples): fix weird .clang-format formatting by adding trailing commas * chore(.clang-format): update for ease of switching between configs * wip(user-agent): add ua_enqueue() for multiplexing requests * wip(test): add test-discord-multiplex.c * fix(user-agent): CURLOPT_POSTFIELDSIZE must be called first than CURLOPT_COPYPOSTFIELDS * docs(websockets.c): typo * chore(common/third-party): add pqueue to and update licenses * style(websockets): '_ws_' function prefixing for consistency * wip: generic worker thread implementation * feat(types.h): add ORCA_GLOBAL_INIT error type * chore(user-agent): remove unnecessary functions * chore: change some comments * feat: check to initialize globals once * wip: replace discord's threadpool with generic implementation * fix(test-discord-ws.c): setenv() should be called before threadpool initialization * chore(work.c): minimum size of 8 slots per threadpool's queue * feat(user-agent): add ua_set_curl_multi() * docs(user-agent.h): document all functions and datatypes * refactor(work.c): remove libpqueue from dependencies and code * feat: add libuv's queue.h * wip(user-agent): queue functions for asynchronous IO implementation * refactor(user-agent): fix premature locking * docs(user-agent): document leftover fields and a couple functionalities * refactor(user-agent): make ua_conn a public opaque datatype, add a field for holding onto user callback and context * chore(common): move queue.h to common/third-party/ * docs: improve explanation for ua_clone() and discord_clone() * refactor(user-agent): move internal _ua_conn_setup() logging to outside, queue functions * feat(discord): add discord_timestamp() * refactor(websockets): move cleanup logic from ws_perform() to ws_end() * chore: update to match 4c3eb14 * refactor(examples): replace cee_timestamp_ms() with discord_timestamp() * refactor(websockets): expose CURLM and CURL handles used internally * feat(websockets.c): replace pthread_mutex with pthread_rwlock where it makes sense * refactor(websockets.c): simplify ws_perform() to return a bool on whether the connection is alive * refactor(websockets): add struct ws_attr for passing optional fields. * feat(websockets): add ws_timestamp_update() for manually updating internally used timestamp * refactor(websockets.c): remove noop functions and simply check for NULL * chore: update to match 656da47 * refactor(user-agent): ua_init() expect a struct ua_attr * feat(user-agent): add ua_timestamp() for last request performed * chore: update to match 87d3110 * docs(user-agent.c): fix comment * fix(discord-ratelimit): don't rely on system time for 'X-Ratelimit-Reset' * refactor(discord-ratelimit): use a shared variable for global ratelimiting * chore(discord): organize and document code somewhat * chore: remove ua_block_ms(), there are better solutions for global ratelimiting (see discord) * fix(user-agent.c): no need to let libcurl create its unique copy for POST fields * chore(third-party): add libuv's heap-inl.h for min-max heap * fix(discord-adapter): JSON error callback being ignored for functions without a resp_handle assigned * wip(discord): use custom CURLM handle for websockets * chore(user-agent): remove unnecessary queue functionalities * feat(user-agent): make ua_conn_get() and ua_conn_setup() public * chore: remove pqueue from codebase * refactor(user-agent): replace connection pool logic with queue.h * fix: undefined behavior mentioned at libuv/libuv#565 * chore(user-agent.c): fix typo * wip(discord): isolate ratelimit logic * feat(discord-ratelimit): all requests will be assigned to a bucket, this will make request enqueing easier (for asynchronous purposes) * fix(discord-ratelimit): memory leak * chore(.clang-format): break before non-assignment operators * fix(discord-adapter.c): major parameter buckets must be decided on a 'id' basis * chore(discord-adapter.c): fix minor typo * chore(discord-adapter.c): correct buffer size * refactor(discord-adapter.c): simplify major param selector * chore(discord-ratelimit.c): null bucket tags should be [null] instead of [?] * refactor(discord-adapter.c): modularize 8a38f37 * refactor(discord-gateway.c): remove repetitive shutdown logging * refactor(discord-ratelimit.c): unnecessary tmp buffer for holding hash * docs(discord-ratelimit.c): add ratelimiting rationale * docs(discord-internal.h): brief explanation for 'struct discord_route' * refactor: prefix singleton buckets with 'b_' * refactor(discord-gateway): simplify event scheduling logic * chore(discord-gateway): remove on_event_raw event because the same can be achieved by the discord_set_event_scheduler() callback * wip(discord-adapter.c): add discord_adapter_enqueue() for asynchronous requests * fix(discord): move request queues to 'struct discord_adapter' * wip(discord-adapter): check for IO polling results with discord_adapter_check() * chore(.clang-format): don't allow single-line for case label * feat(discord-ratelimit.c): add discord_route_get() and improve docs for discord_bucket_get() * wip(discord-adapter.c): asynchronous IO request enqueueing logic * refactor(discord-adapter.c): modularize code that should be used in both sync and async scenarios * fix(discord-adapter.c): wrong info * feat!(user-agent): replace request timestamp with libcurl provided elapsed time, remove ua_timestamp() * refactor(user-agent.c): move 'post-request' logic from _ua_conn_send() to _ua_conn_check_status() * refactor(user-agent.c): rename _ua_conn_reset() to ua_conn_stop() and make it public, rename ua_conn_get() to ua_conn_start() * chore(js_user-agent.c): match fad5f changes * refactor(discord-ratelimit.c): checking for bucket's update timestamp is unnecessary * refactor(discord-internal.h): 'struct discord_request_cxt' stores 'struct ua_conn' * docs(discord-internal.h): more descriptive documentation for 'struct discord_request_cxt' * chore(discord-internal.h): get rid of bucket.update_tstamp * feat(discord-adapter.c): get and assign 'struct ua_conn' to the request handler (4768962) * feat(user-agent): add discord_conn_get_results() * refactor(user-agent.c): simplify code, get rid of compound literals and repetition * chore(js_user-agent.c): update to match 6eb0740 * fix(user-agent.h): declare ua_conn_get_results(), change docs regarding 'info' parameters * wip(discord-adapter.c): make async layout similar to _discord_adapter_request(), except queues * wip(discord): discord_set_async() for performing requests asynchronously * wip(discord-adapter.c): add a couple TODOs * chore(discord): rename 'discord_request_cxt' to 'discord_request', and 'discord_event_cxt' to 'discord_event', shorten a couple fields * style(discord): run latest .clang-format for discord-gateway.c and discord-client.c * refactor(user-agent): remove redundant callbacks (use ua_info instead) * chore(discord-gateway.c): rewrite compound literals and move variables declaration to top * refactor(discord-client.c): discord_timestamp() shall use ws_timestamp() only if there's a live connection * fix(discord-adapter.c): struct discord_request should hold a bucket field, missing http method, and route field * wip(test-discord-async.c): test first prototype * fix: _ws_curl_tls_check() should only trigger ws_close() if the user hasn't done himself * wip(discord-adapter.c): async trigger user callback, use discord_timestamp() * refactor(discord): share only whats necessary, shorten a couple fields * refactor(discord-gateway.c): apply 9220b changes, add close reason, fix reconnect logic * fix(discord-ratelimit.c): shouldn't skip discord_bucket_build() on unsuccesful requests * chore(discord-voice-connections.c): apply 9220b changes * feat(test-discord-ws.c): add reconnect test" * feat(test-discord-async.c): test asynchronous vs synchronous * feat(discord-adapter.c): queues cleanup logic * feat(discord-ratelimit.c): support out-of-order ratelimiting (for multiplexing reasons) * chore(test): remove test-discord-multiplex.c * wip(discord): request timeout logic * refactor(discord-ratelimit.c): split a couple functions * fix(discord-gateway.c): send CLOSE_REASON_NO_REASON if a reconnect attempt is to follow * refactor(discord-gateway.c): replace cee_timestamp_ms() with ws_timestamp() to reduce OS calls * refactor(discord-voice-connections.c): match discord-gateway.c logic * chore(bot-elitebgs.c): update to latest * refactor(user-agent.c): add _ua_info_reset() and _ua_info_populate() * feat(discord): functional ratelimiting for async * chore(test-discord-async.c): update test * chore(discord): rename discord-ratelimit.c to discord-adapter-ratelimit.c * refactor(discord-adapter-request.c): split request handling logic from discord-adapter.c and discord-adapter-ratelimit.c * refactor(discord-adapter-request.c): consistent naming and descriptive comments * feat(user-agent): add ua_conn_reset() * feat(discord-adapter-ratelimit.c): add discord_bucket_get_timeout() and remove discord_bucket_timeout() * feat(discord-adapter-request.c): timeout on non-global 429 * fix(discord-adapter-ratelimit.c): in case multiple requests with undefined buckets are enqueued at once, sort them out at once the first one gets a bucket match * fix(discord): ratelimit per-route, not per-hash * feat(discord): set priority of async request * chore(test-discord-async.c): update * fix(user-agent.c): move HTTP_SEND logging to ua_conn_setup() so that it activates for asynchronous requests * feat(discord): stop on-going requests with discord_request_stop_all() * fix(user-agent.c): typo us_conn_perform() -> ua_conn_perform() * fix(discord-adapter-request.c): req_body recycling logic, memsize should be kept separate from length * chore(test-discord-async.c): add ordered spamming test * refactor(discord-adapter-ratelimit.c): alter a couple logging levels * fix(discord-adapter-request.c): don't force decrease remaining buckets value * fix(discord-adapter-ratelimit.c): skip when current timestamp is lesser than reset timestamp * fix(discord-gateway.c): no need to run request checks if connection has been severed * refactor(discord-internal.h): remove unused 'server' field * feat: added discord_set_on_commands (#750) * feat(examples): add simpler slash-commands example * feat(specs/discord): missing fields for 'Application Command Options' * chore(specs-code): update to match latest 807c590 changes * feat(bot-slash-commands.c): update with channel listing example, rename a couple fields * refactor(bot-slash-commands.c): simplify input read * feat: added discord_set_on_commands * docs: added docstring for discord_set_on_commands Co-authored-by: lcsmuller <[email protected]> * chore(discord-adapter-ratelimit.c): reduce scope * refactor(discord-gateway.c): add _discord_gateway_close() * fix: freeze bucket on any request timed out, unfreeze after it time out has passed * chore(test-discord-async.c): asynchronous infinite spamming * style(discord-channel.c): ANSI * feat(specs/discord/gateway.json): add DISCORD_GATEWAY_CLOSE_REASON_RECONNECT enumerator * chore(specs-code): update to match 6618c1a * fix(discord-gateway.c): use opcode 4900 for reconnect * fix(user-agent.c): curl_mime_free() should be called on conn's stop * refactor(websockets): ws_start() initialize the multi handle, ws_end() cleans it up * chore: update to match 84f0a9a * feat(discord-adapter-request.c): add discord_request_pause_all() for pausing and resuming async transfers * wip: support ANSI syntax * fix(discord-emoji.c): wrong parameter address * chore: delete stale/redundant/unused files * chore(test-slack-ws.c): move from 'tmp' * fix(test-slack-ws.c): incompatible callback signature * refactor: shorten user-agent.c symbols * refactor(user-agent): consistency with function naming * feat(discord-adapter-request.c): if set, async callback receive response body for parsing into object * refactor: improve naming consistency and get rid of unintuitive NTL_T macros * refactor(discord-adapter-request.c): change discord_async_cb signature and update to match 5af0a43 * fix(discord): idle queue must be heap-allocated to ensure its shared between original and cloned clients * docs(discord): move 'Event Scheduling' functions into its own doxygen category * refactor(discord): move discord_set_async() to discord-adapter.c and rename it to discord_adapter_set_async() * wip(discord-internal.h): for ANSI compliancy anonymous structures should be replaced * feat(discord-adapter-request.c): use recycleable buffer for callback return object * refactor(discord): reorg * feat(discord): add discord_create_message_async() * chore(test-discord-async.c): update to match a066ea0 * fix(discord-adapter-ratelimit.c): obtaining length of major parameter * refactor(user-agent): remove ua_reqheader_del(), make code ANSI C compliant * style(common): ANSI * wip: stackful ua_conn * refactor(common): rename ORCA_NO_RESPONSE to ORCA_CURL_NO_RESPONSE * fix(discord-adapter-ratelimit.c): UB for types with system-dependent sizes * fix(discord-adapter-request.c): 3fcdae and reset 'conn' values after each usage, retry on ORCA_CURL_NO_RESPONSE * fix(websockets.c): don't free multi handle at ws_end(), in case we might be reconnecting * fix(discord-adapter-request.c): enqueue again on read error, build bucket on success * chore(test-discord-async.c): update * refactor(user-agent): rename ua_conn_get_results() to ua_info_extract(), remove struct ua_resp_handle parameter from ua_conn_setup() * feat(discord): replace struct ua_resp_handle with the more flexible struct discord_request_attr * chore(test-discord-async.c): match ae9ed * fix(bot-fetch-messages.c): dereferencing null-pointer * feat(user-agent): struct ua_info stores the request error code * refactor(discord-internal): move 'struct discord_adapter' components that makes more hierarchical to 'struct discord_ratelimit' * refactor(bot-shell.c): simplify and use async * chore(.clang-format): add comment * feat(discord): enable sending files over the multiplexer * chore(bot-shell.c): redundant check * docs(discord.h): improve discord_set_on_command() and discord_set_on_commands() * refactor(bot-shell.c): move attachments outside of embed and use discord_set_on_commands() * style(bot-shell.c): .clang-format * feat(types.h): add CONTAINEROF() macro * refactor(websockets.c): remove misleading error * refactor(discord): rename 'discord_request' to 'discord_context' * refactor(discord): rename 'discord_ratelimit' to 'discord_request' * feat(discord-internal.h): add discord_bucket_init() and discord_buckets_cleanup() * refactor(discord-adapter-request): concentrate otherwise scattered 'discord_request' logic over * feat(discord-internal.h): add generic CLIENT() macro that wraps around CONTAINEROF() * fix(discord-gateway.c): when a transfer is complete, its properly checked and acted on if its websockets on REST related, instead of just ignoring * fix(discord-adapter-request): recycleable buffer is assigned to cxt->attr.obj when first enqueueing request at _discord_context_populate() * refactor(discord-adapter-request): replace discord_request_check_results_async() with discord_request_check_action() * chore(discord): remove all discord_sb_* references * chore(discord-internal.h): reorganize structures hierarchichally * refactor(types): add ORCA_EXPECT() macro, remove ORCA_MISSING_PARAMETER in favor of ORCA_BAD_PARAMETER * refactor: make use of ORCA_EXPECT() to improve consistency and readability * refactor(discord): replace exposed 'ja_u64**' type with 'u64_snowflake_t**' * chore(bot-echo.c): simplify * chore(bot-echo.c): should be discord_create_message_async() * chore(discord-user.c): remove sb_discord_get_current_user() * chore(test-discord-async.c): match to most recent * fix(discord-channel.c): UB when iterating over a list without checking for NULL * refactor(websockets): rename cws_reqheader_add() -> cws_add_header(), ws_reqheader_add() -> ws_add_header() * refactor(websockets): remove unused logic to keep code minimal * refactor(discord): keep ws_timestamp_update() value stored to save on system calls * fix(discord-gateway): 'now' timestamp must be shared between copies * feat(discord): use discord_timestamp() only on MT-Unsafe portions * refactor(discord-gateway): merge reconnect structure to status, rename it to retry * refactor!(discord): remove bot parameter from callbacks, instead call discord_get_self() * refactor(discord): rename callbacks * chore(discord): consolidate 2140f changes * fix(user-agent.c): UB on _ua_conn_respheader_cb(), strings aren't null-terminated, shouldn't use string.h functions * chore(discord-adapter-request.c): remove leftover asserts * fix(user-agent.c): skipping extra char * fix(discord): remove redundant const specifier on callback scalars * chore(examples): update to match e14bca * feat(websockets): enable CURLOPT_NOSIGNAL, more descriptive errors, log instead of crashing on error * fix(websockets): macro typo * docs(discord-adapter-request.c): add relevant TODO * refactor!(user-agent): rename ua_run() to ua_easy_run() and add disclaimer * chore: match e291b * fix(discord-adapter.c): fallback to read-only blank attr in case of NULL * refactor(github): make it consistent with discord codebase * fix(bot-elitebgs.c): use ua_easy_run() * fix(test-cee.c): use ua_easy_run() * added functionality for guild_on_(create/update/delete) (#751) * refactor(discord-adapter): rename discord_bucket_cooldown() to discord_bucket_get_wait() and return sleep time * fix(discord-adapter-request.c): update 'now' timestamp before populating bucket in a blocking request * chore(test-discord-ws.c): test ratelimiting on single-threaded blocking request * refactor(discord): join 'discord_gateway' .status field to .session, use bitmask for gateway status * refactor(discord): move bot structure (struct discord_gateway -> struct discord), move identify and event timestamps * chore(common): unnecessary do while (0) wrapping over macros * refactor!(websockets): add extra field for obtaining timestamp, rename ws_perform() to ws_easy_run() and add disclaimer at its documentation * chore: match 74543 * fix(discord-gateway.c): wrong bitwise op * refactor!(websockets): rid of ws_start() last arguments, ws_init() will expects a user-owned curl_multi handle, rid of compound literals * chore: match 22aaf * wip(discord): merge struct discord_request to struct discord_adapter * refactor(discord): finish transitioning 'discord_request' references to 'discord_adapter' * chore(bot-voice): include discord-internal.h instead * wip(discord-gateway.c): disable possibly wrong ratelimiting logic * refactor(discord-gateway.c): simplify on_dispatch (#752) Co-authored-by: Lucas Müller <[email protected]> * fix(discord): discord_adapter_pause_all() is unnecessary after decoupling websockets and requests logic * chore(discord-gateway.c): rollback from #752, my bad @Anotra * chore(cee-utils): get latest * fix(discord): incomplete type * feat(user-agent): add 'struct ua_conn_attr' for setting connection attributes, to improve flexibility when adding new config * refactor(user-agent.c): simplify _ua_conn_set_url() * refactor: match codebase to 9310c * chore(slack, github, reddit): move request functions to single file * chore(specs): shorten naming * chore(specs-code): match latest * chore: reorg * chore(specs): generate params for slack * chore(specs-code): update to f02e74 * fix: undefined retry value * feat: rename types.[c|h] to common.[c|h], add orca_global_init() and orca_global_cleanup() * chore(discord): match to d697d * refactor(js_user-agent): rename jsua_run() to jsua_easy_run() * fix(discord-gateway.c): scheduler callback shouldn't be skipped * refactor!(slack): make it consistent with the codebase * chore: move initialization macro to single-file where it should be used * chore(discord): merge request functions into a single file * chore(discord): rename file * refactor: move macros to discord-restapi.c * refactor(discord-misc): move discord_disconnect_member() to discord-restapi.c * chore: . * feat(discord): make discord_async_next() public * chore(discord): match to db547 * refactor(examples): replace deprecated discord_global_init() and discord_global_cleanup() Co-authored-by: antropez <[email protected]> Co-authored-by: Anotra <[email protected]>
- Loading branch information
3 people
authored
Dec 18, 2021
1 parent
9d34a8c
commit 96e3674