Bug Report | Performance issues?

[Fastechgaming]

Information

After installing the plugin and reaching 30+ players online, my server started using a lot of CPU, almost hitting 100%. This caused players to experience high ping and frequent false flags on the server. Here's the spark profile: https://spark.lucko.me/iyfguGitVB

When does the bug occure?

When players online 30+

Cardinal Version

5.1.1-alpha-nightly-efacf8d2

Spigot Version

1.20.4

Additional information

No response

[micartey]

I tested the plugin recently with around 120 players which worked fine on 1.21.

There are several things you could try:

• What server version are you using?
• Use more performant GC settings
• decrease the tracker amount in the Config.yml to 1 (Cardinal)

While I had no issues running 120 players, I didn't have ~500 chunks loaded.

As a last resort you can look into increasing netty threads. The lags are not due to falling TPS, but congestion in the netty threads. Increasing the amount might help, but I am unsure how Minecraft handles the amount of threads.

Apart from Minecraft it might also be interesting to know what server resources you have. A slow cpu and/or memory might be disadvantageous as well

[Fastechgaming]

I'll try "decrease the tracker amount in the Config.yml to 1" but how to "use more performant GC settings" i don't what it is.
And my server is just a boxpvp server. No chunk generate needed and no mobs can spawn.

Also, My cpu when almost 100% after installing the plugin tho and I think you can check my spark and see my server resources. Here: https://spark.lucko.me/iyfguGitVB

[micartey]

Spark is not very good at showing issues apart from normal bukkit thread utilization. Can you report a async-profiler report for me? This is a lot better to debug and understand possible bottlenecks

[Fastechgaming]

Is "async-profiler report" a built-in paper? Or a the top one when I search up "async-profiler"

[micartey]

The latter.

It is an external tool for the JVM

[Fastechgaming]

It's gonna be so hard. Since I don't know about it at all

[micartey]

1. Run wget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-linux-x64.tar.gz
2. Run tar -xvzf async-profiler-3.0-linux-x64.tar.gz
3. CD into the extracted folder (it should be named sth like async-profiler-3.0-linux-x64)
4. CD into bin
5. Grant execute permissions: chmod +x asprof
6. Figure out the server pid. You maybe can do so using ./asprof list but this doesn't seem to work for me
7. Run ./asprof -d 300 -f report.html <PID>

Send me the generated report

[jonesdevelopment]

Try to see if /spark profiler start --thread * shows more information.

[micartey]

While that might be true, it certainly doesn't show more information than async profiler. But we can try first

[Fastechgaming]

I don't think I can get the "async-profiler". Because I'm using a Minecraft hosting. Not VPS or smth.

[jonesdevelopment]

Try to see if /spark profiler start --thread * shows more information.

@Fastechgaming Can you try this and post the new spark report here?

[Fastechgaming]

https://spark.lucko.me/27CHFz1Z5u
Here. My CPU eat up so much.

[Fastechgaming]

https://spark.lucko.me/pnGWkaO9tH
This is during the lagg. Only when I install CardinalAnticheat.

[micartey]

Both reports have cardinal installed?

I don't quite know how to read that, but it seems that packet events being used twice may be the main issue.

What do you think @jonesdevelopment ?

[jonesdevelopment]

[jonesdevelopment]

Your EventManager is taking up >30% of the Netty thread. @micartey

[micartey]

That is wrong 👀

[micartey]

Ah, now I understand. It is not the percentage of the percentage...

[micartey]

I am currently not seeing any optimizations I could make.

While the Event System is the bottleneck, this is only the case because there is the entire invocation of the checks and stuff.
There is also the Packet Events plugin that has roughly the same usage.

I think the main issue is not the event system, but that you use 3 anticheats.
Anticheats are heavy, need to do a lot of calculation and if you want to be able to mitigate players, you can't really be async from the netty thread.
This leads to congestion and the netty thread is too slow.

I tested Cardinal with 100 players and it worked fine.
But you have 3-times the workload...

[Fastechgaming]

I see. Well it is fine when I use Themis as bedrock anticheat. I'll remove Vulcan for now. Since I use it for Aim check only.

[Fastechgaming]

I have remove my 2 other anticheat and only use your. Everyone on my server say the server lag badly. Like after 1-3mn lag for 10s. CPU usage updown updown.
Here the spark: https://spark.lucko.me/bobL3oJ5kk

And this one is without Cardinal Anticheat plugin but with my 2 other anticheat. Everything is fine.
https://spark.lucko.me/0lKTTzPOdA

Both have the same 1h55mn uptime(I didn't realise that) also around 60players.

Here's 90players online without any lagg using my 2anticheat.

https://github.com/user-attachments/assets/033b529e-42b5-4bdc-9c8c-ac649e73ff83

[Fastechgaming]

Also, when it's lag sometimes. It flags everyone badly. 30% off them getting kicked because of false flags

https://github.com/user-attachments/assets/8a2a8354-1413-48c3-8558-42ca2ab61d92
https://github.com/user-attachments/assets/a2a72aab-7ef1-4a55-a657-0b794a03f276

[micartey]

I made some smaller optimizations, but I am not sure how big the impact would be.
I simply don't have the possibility to test and reproduce your issues so you would need to tell me what works and what doesn't.

Did you decrease the tracker amount as mentioned previously?

[Fastechgaming]

I did set it to 1

[micartey]

Did you try it with the newer version (simply restart your server with Cardinal)

[Fastechgaming]

latest (6).log
The plugin not loaded

[micartey]

Should work now

[Fastechgaming]

latest (7).log
Can you check this? Something spamming in console from Cardinal.

[Fastechgaming]

2025-01-31-6.log

My Server just Crashed.

[micartey]

I pushed a fix. Can you try again?

It is currently only a workaround but when this solves the exceptions I at least know where to look at.

[Fastechgaming]

[02:08:40 ERROR]: Could not pass event PlayerQuitEvent to Cardinal v1.0.0
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long (java.lang.Integer and java.lang.Long are in module java.base of loader 'bootstrap')
at me.clientastisch.cardinal.dependencies.fastutil.longs.Long2ObjectFunction.remove(Unknown Source) ~[?:?]
at me.clientastisch.cardinal.dependencies.fastutil.longs.Long2ObjectMap.remove(Unknown Source) ~[?:?]
at me.clientastisch.cardinal.bo.d(Unknown Source) ~[?:?]
at me.clientastisch.cardinal.bo.e(Unknown Source) ~[?:?]
at me.clientastisch.cardinal.events.bukkit.BukkitQuitEvent.onQuit(Unknown Source) ~[?:?]
at com.destroystokyo.paper.event.executor.asm.generated.GeneratedEventExecutor387.execute(Unknown Source) ~[?:?]
at org.bukkit.plugin.EventExecutor$2.execute(EventExecutor.java:77) ~[purpur-api-1.20.4-R0.1-SNAPSHOT.jar:?]
at co.aikar.timings.TimedEventExecutor.execute(TimedEventExecutor.java:77) ~[purpur-api-1.20.4-R0.1-SNAPSHOT.jar:git-Purpur-2176]
at org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:70) ~[purpur-api-1.20.4-R0.1-SNAPSHOT.jar:?]
at io.papermc.paper.plugin.manager.PaperEventManager.callEvent(PaperEventManager.java:54) ~[purpur-1.20.4.jar:git-Purpur-2176]
at io.papermc.paper.plugin.manager.PaperPluginManagerImpl.callEvent(PaperPluginManagerImpl.java:126) ~[purpur-1.20.4.jar:git-Purpur-2176]
at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:617) ~[purpur-api-1.20.4-R0.1-SNAPSHOT.jar:?]
at net.minecraft.server.players.PlayerList.remove(PlayerList.java:615) ~[purpur-1.20.4.jar:git-Purpur-2176]
at net.minecraft.server.players.PlayerList.remove(PlayerList.java:599) ~[purpur-1.20.4.jar:git-Purpur-2176]
at net.minecraft.server.network.ServerGamePacketListenerImpl.removePlayerFromWorld(ServerGamePacketListenerImpl.java:2164) ~[?:?]
at net.minecraft.server.network.ServerGamePacketListenerImpl.onDisconnect(ServerGamePacketListenerImpl.java:2144) ~[?:?]
at net.minecraft.server.network.ServerGamePacketListenerImpl.onDisconnect(ServerGamePacketListenerImpl.java:2131) ~[?:?]
at net.minecraft.network.Connection.handleDisconnection(Connection.java:847) ~[?:?]
at net.minecraft.server.network.ServerConnectionListener.tick(ServerConnectionListener.java:254) ~[?:?]
at net.minecraft.server.MinecraftServer.tickChildren(MinecraftServer.java:1781) ~[purpur-1.20.4.jar:git-Purpur-2176]
at net.minecraft.server.dedicated.DedicatedServer.tickChildren(DedicatedServer.java:487) ~[purpur-1.20.4.jar:git-Purpur-2176]
at net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1558) ~[purpur-1.20.4.jar:git-Purpur-2176]
at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1246) ~[purpur-1.20.4.jar:git-Purpur-2176]
at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:323) ~[purpur-1.20.4.jar:git-Purpur-2176]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]

This spam in my console

[micartey]

Should be fixed