Accounts panel & UserInfo RCEP rework #140
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LoadAll
&LoadRegistered
to return Dictionary with UID as keys. Few places actually used these, pretty much just in Frontend stuff.With both of these, can avoid checking all many thousand
BasicUserInfo
for uncluttered accounts panel this way.New RCEP
/api/userinfosfiltered
takes params:onlyspecial={true|false}
to only return the tagged users from cached Dict, and fetch BanInfos and KickHistorys, currently it returns 458 accounts/User infos (optional, default false)forcereload={true|false}
to refresh the mentioned dict of tagged users from database (optional, default false)from={first}&count={count}
same as in existing/userinfos
EP (optional, default 0 - 100)search={search}
filter names, ignores case (optional)Specifically the change to UserData is this:
(plus changes to the implementations of course)
TODO:
Overview of the issue and timings
With the very scientific method of "where do I have to place the scroll bar to see certain ranges of durations" I can see that all the userinfos EP updates that happen on the actual server on sans take on average 9 seconds, +/- 1.
Some take as long as 17 seconds.
(I'm currently also questioning if we should stop making the websocket trigger these automatically entirely?... Since I'm pretty sure it was broken for the longest time anyways.)
Running the server locally on my PC, it takes almost half a minute to fetch the entire 7+ MB of userinfos. I'm surprised how bad this is.
PS: Oh yeah and when switching to "cluttered" view on current main build, we know that it just gives the browser a heart attack because it adds 40k entries to the panel list. Including the ~25s it takes about 80 to 90s for my Firefox tab to recover from that ordeal at the moment.
Running with this PR, it defaults to the new endpoint, which will cache "tagged" users in memory:
This takes about 7 seconds at startup for me, and can also be triggered by the EP on demand, but shouldn't be necessary all that often.
The other categories of "uncluttered"/special users would be kicks and bans, which can already be loaded efficiently since the respective serialized data blobs already contain the UIDs and don't require going through the entire database of BasicUserInfos.
This is how long requests take with server-side filtering, where pagination will only fetch slices of 200 accounts at a time. The downside to this is that currently there's no good way to page through a list of accounts sorted by name or anything, since that would take the 10 or 25 second penalty.
Letting SQLite sort them is not viable either because the names and such are stuck in the serialized blobs (we're essentially running more of a MongoDB type document storage than anything even remotely relational)
Or you can use the "in browser" filtering mode, where you HAVE to manually invoke the 10s / 25s full fetch of userinfos once but then you can filter and sort near-instantly in browser.
The tooltips of what the buttons do:
The text shown is always what it's currently set to. The "Filter Mode" toggle changes the meaning of the Reload/Refresh button and associated logic:
The "Filter" of the third button was previously "unclutter/clutter" with the visibility icon. It is now "Filter: Show All" for the "cluttered" view and "Filter: Kick/Ban/Tag" for "uncluttered".
Again for the same reasons stated as for sorting, the server-side filtering option currently does no calculations for how many total results your search may return. So pagination gets a bit silly.
E.g. for the 458 "filtered" accounts, you get pages 2, 3, ...:
Where it simply fetches "from=200&to=400" and so on, and on page 4 and beyond there are no more results to be found.
I don't see this as a huge issue tbh and currently have no plans to fix this unless we overhaul UserData modules tbh 🧌
In "in browser" filter mode you get correct total pages at least but you can still go beyond because idc to fix :3