Admin
Owner

9 posts
5 reputation

IGN: Admin
By Admin » 8 months ago
I find reading the Hypixel dev blog very interesting, so I thought I'd write about our internal systems for those who are interested.

First off, I'd like to apologise for the down time to the website and the server yesterday (Saturday) due to these upgrades. So, let's get started!

Let's start with a little context: for Perkelle, we have a dedicated server from SYS with a 4790K and 32GB of RAM, which is enough to run our game servers for now. In addition to this, we have a secondary VPS from OVH (although we're looking to move that to Scaleway) which we host our lightweight backend microservices on so that they have minimal downtime. These microservices include our new Zeus system, our VPN checking system and plugins.

I also have a VPS of my own with a Threadripper 1950X elsewhere that I run Perkelle Bot on and a Gitlab runner for our instance.

Zeus is a our new microservice that is responsible for storing and distributing server info. I originally started writing in Golang for maximum performance, however, I later switch to Kotlin as I wanted to experiment with Ktor, a Kotlin HTTP framework, and also I was able to do development faster as I know Kotlin a lot better than Golang. I opensourced the original version, the code can be viewed on GitHub here: https://github.com/Dot-Rar/MasterController

Next, I wrote Ares, a client for Zeus that is able to run on BungeeCord and on Spigot servers. This adds a layer of abstraction so that our other plugins are able to easily interact with Zeus. I originally started by using KHTTP as the HTTP client library, but later switched to Fuel, and finally Apache Fluent HC. You can see an example of how the internal Ares code looks for connecting to one of the endpoints here:
 
class GamemodeTotalCount(val pl: MultiPlatformPlugin) : Endpoint() {

    override val endpoint = "/gamemode/total"

    override val requestType = RequestType.GET

    fun execute(gamemode: Gamemode, callback: Callback<int>) {
        pl.async {
            try {
                val res = request<response>(queryParams=*arrayOf("gamemode" to gamemode.internalName))

                if(!res.pojo.success) {
                    pl.log("GamemodeTotalCount failed with error code: ${res.pojo.error_code}; Gamemode: $gamemode")
                    pl.wait(2) {
                        execute(gamemode, callback)
                    }
                    return@async
                }

                callback(res.pojo.total!!)
            } catch(ex: Exception) {
                ex.printStackTrace()
                pl.wait(2) {
                    execute(gamemode, callback)
                }
            }
        }
    }

    data class Response(val success: Boolean, val error_code: Int?, val total: Int?)
}</response></int>
This then allows me to make simple method calls to retrieve data from Zeus.

So, when a Spigot server starts, the following process takes place:
- Contacts Zeus and reports the gamemode, hostname and port of the server.
- Zeus assigns the Spigot server a UUID
- The Ares plugin sends a heartbeat to Zeus every 15 seconds

Then, every 5 seconds, all BungeeCord instances:
- Retrieve a list of servers and all data from Zeus
- Remove dead servers (not sending heartbeat) from the BungeeCord server list
- Add new servers to the BungeeCord server list

And that's pretty much it. It's quite a simple process in reality, however, it opens up a lot of options for us. Having an external microservice controller means that in the future, we are able to scale a lot easier, and also handle inter-server communication without the crude use of Redis pubsub. 

Please let me know if you're interested in me writing more of these in the future, and what you'd like to see!