Google Summer of Code 2017 final report

Usage instructions

Server

• Install Docker (if it’s not already installed on the machine you want to use). From this page there are instructions for the various operating systems and cloud computing platforms.
• In your shell, execute docker pull gianlucanitti/terasologyfacadeserver. This will download the binary image and may take some time.
• When the docker pull command returns, you can start the server with docker run -p 8080:8080 -p 25777:25777 -d gianlucanitti/terasologyfacadeserver. A couple of notes:
  1. The two -p arguments expose the network ports for the web server (8080) and for the Terasology game server (25777) to the host machine (and the external world, if allowed by your firewall). Without them, it wouldn’t be possible to connect to the server.
  2. The -d switch is optional. If present, the server will start detached from the shell so that you can leave the shell/terminal without stopping it.
• If you used the -d switch, the command will pring a long string which is the ID of the started Docker container and return. You can then view the server’s log with docker logs <container-id> or stop it with docker stop <container-id>, replacing <containder-id> with the string printed by the docker run command. If you started without the -d switch, the server’s log will be printed to the standard output, and the server can be stopped with CTRL+C.
  
  Note that the Docker image is packaged without any additional module, so no default game is started and you cannot directly connect to the server using the regular game client; you first need to connect to it using the web interface, install the modules you want to use and start a game.
  
  Alternatively, you can run it from a development environment. In this case, you need to clone the main Terasology repository, then obtain FacadeServer’s source by running ./gradlew fetchFacadeServer and switching to the develop branch of FacadeServer by running cd facades/Server && git checkout develop && cd ../... At this point, you can use ./gradlew jar facades:Server:run to compile everything and start the server.

Client

Code

MovingBlocks/FacadeServer

• Run the game engine (merged)
• Authentication management (merged)
• Resource management (merged)
• Games resource (merged)
• Modules install resource (merged)
• Add resources to obtain or modify server’s MOTD and port (merged)
• Manage administrative permissions (merged)
• REST methods (open)

MovingBlocks/Terasology

gianluca-nitti/FacadeServer-frontend

Other

• The commits after 1633d18 on the master branch of my terasology-key-server repository, which is a tool to help users in carrying their Terasology client identity certificates across various clients (a running instance is availavble here), has been done during the GSoC work period to adapt it to be used in the web and mobile application too (in addition to the Terasology game client).
• I made this very small pull request to the MovingBlocks/meta-server respository.

Other relevant links

• Development blog website
• GitHub project board

Demonstrational video

Thanks and credits

Improved mobile client user experience and implemented a more RESTful-compliant HTTP API on the server

• GET /onlinePlayers -> get online player list
• POST /console -> run command specified in payload
• GET /games -> get savegame list
• GET /games/{name} -> get details about a single game
• POST /games -> create new game (properties specified in payload)
• PATCH /games/{name} -> rename game with new name specified in payload (possible addition: change modules)
• DELETE /games/{name} -> delete game
• POST /games/{name}/backup -> create a backup
• GET /engineState -> get the engine state
• PUT /engineState -> switch to state (payload contains new state, can be IDLE or LOADING, and game name if necessary)
• GET /modules/available -> get list of installed modules
• GET /modules/available/{name} -> get details about single module
• DELETE /modules/available/{name} -> uninstall specific module (currently not implemented; what about dependencies?)
• GET /modules/installer -> get module installer status message
• PUT /modules/installer -> start installing a set of modules specified as payload
• GET /worldGenerators -> get list of available world generators
• GET /config/serverPort -> get listen port
• PUT /config/serverPort -> change listen port
• GET /config/MOTD -> get MOTD
• PUT /config/MOTD -> change MOTD
• GET /serverAdmins -> get list of admins
• POST /serverAdmins/{clientId} -> add client ID to the admin list
• DELETE /serverAdmins/{clientId} -> delete client ID from admin list

Completed admin permission management, added automatic backups to identity storage server, and added a new feature to the frontend

• The list of users allowed to perform admin-level operations consist in a list of the client IDs of said users, and is stored in a serverAdmins.json file in the server data directory (next to the config.cfg configuration file).
• When a server is started for the first time (i.e. the data directory specified on the command line does not exist or is empty, so it’s created and the server configuration is initialized to the default values), the administrator list will be empty. Also, since there are no savegames, no regular client ever connected, so no client identities have been generated at all. When the list is empty, everyone which can connect to the server using the HTTP/WebSocket API also has access to the admin-restricted features (there would be no way to identify a client in this initial phase).
• After starting a new server, it’s recommended to lock the administrative access to the trusted user(s); this can be done by creating a game from the web interface and starting it, then connecting to the server using the regular game client. When the server is in the “empty admin list, public access” phase, in fact, the first user which connects will be added to the admin list. Since the list becomes non-empty, this means that from this moment the access to the admin-only functionalities is no longer public, but restricted to the user which joined the server for the first time. Said user then needs to authenticate to the web interface (by pasting the game client’s configuration file or using an identity storage server, as described in older posts on this site) to access these features (so, for example, to add other games, installing other modules, etc.).
• An authenticated user whose ID is in the administrator list can then add other users to the list, to grant them access to the restricted features, or remove them to revoke the permission. Using the proper panel in the web/mobile client, adding a user to the admin list can be done either by manually pasting their client ID in the interface, or, if they are online, by picking their user name from a list.
• If the admin list becomes empty again (which can only happen if an admin removes everyone including himself from it, or if the serverAdmins.json file is deleted and the server process restarted), the access to the admin-only features becomes public again, until a player joins through the regular game client as described above.

Completed module installation from web interface, added access to two configuration entries, and did some improvements on the identity storage server

Completed login via identity storage service and prepared the “module installation” feature

Various small improvements to both client and server

Server

• It’s now possible to rename existing savegames.
• The class which provides information about available modules when creating a new game, AvailableModulesResource, is no longer a proxy for the meta server but shows only the modules installed on the server and also provides information about the available world generators.
• All the server-side code described in this an the previous blog updates is on a new pull request.

Client

• There is a new screen with an animation which is shown when the client is trying to connect to the server, and if the connection fails the dialog which prompts for a server to connect to is shown again; this prevents the user to interact with carious controls before the connection is estabilished which could bring the user interface in an inconsistent state.
• It’s now possible to save addresses of favorite servers in the browser or device to avoid having to type them every time.
• In the dialog to create a new game, the world generator can now be chosen from a drop-down list (you no longer need to manually type the name). Also, the module list now only shows only the modules installed on the server; a separate panel to install/uninstall modules will be added later.
• Some parts have been refactored to improve the codebase (various TODOs have been removed) and the support for Android. This required an update of the ReactXP library to it’s latest version, which also introduced a small visual bug with text fields borders; I reported it and it should be fixed soon.
• Finally, I did some steps to allow authentication via an identity storage service, the feature is not available yet but it required to manually build two additional ReactXP controls.

Added support for various interactions with savegames

• Start and stop games - if a game is running, starting another one will automatically stop the running game, while manually stopping a game will bring the engine to an idle state until another game is started
• Add a new game, by specifying the title, seed, list of modules and world generator to use
• Delete an existing game
• Generate a backup of a game - useful before doing “dangerous” operations like adding or removing modules from an existing savegame, which will be implemented soon
  
  Here is a screenshot of the “New Game” dialog of the web frontend application:

Implemented read-only access to savegames and modules

• A key feature of Terasology is modularity, so the web interface should allow server administrators to add/remove or enable/disable the gameplay modules.
• A Terasology instance can manage multiple savegames (though only one is running at a given moment), each of them with potentially different sets of modules enabled. At the moment, this is mostly used in the “client” mode, where a user can have various single-player savegames; the headless server implementation (see StateHeadlessSetup.java) creates a single savegame with a certain module list, which can be customized by editing the configuration file.
• To enable or disable modules on a certain savegame, that game must not be running. Also, usually the module list is set when the game is created and then left unchanged; I still have to investigate whether it’s possible to add or remove modules on an existing savegame, it probably depends on the module.
• The permission system is internal to each savegame, i.e. a client identified by a certain client ID can have different permissions in different games and there isn’t a “global” permission system.
  
  These considerations led me to the following conclusions:

• Two different resources should be added, one to manage the various savegames and their modules (I’m working on this one at the moment) and one to manage “global” server settings (such as the network port).
• A new engine state which is not running any game is required to allow the server administrator(s) to configure modules for each game. When the engine is in this state, the administrator can create one or more games in the web interface and choose one to run.
• To determine whether a client is a server administrator, a list of the client IDs allowed to perform these restricted task could be saved in a file in the server data directory.
• A command line switch can be used to tell the server if, at first startup, it should generate a game with the default modules or wait for a connection from the web interface to let the administrator manually configure the game, then start it. In a similar way, another command line argument should allow to select which game sould be started automatically if there are multiple ones.
• To allow the administrator to configure a game on the first startup, an additional authentication method needs to be added. In fact, on the first startup, there are no client identities stored in the server configuration file, because they are generated when an user connects through a normal client (not the web application). A simple solution could be to specify a password on the command line or as an environment variable when starting the server instance for the first time, which could then be entered in the web interface.
  
  At the moment, I implemented read-only access to the server status and the game list with module information (both the server-side, on the new games-resource branch, and the client-side, in the frontend repository). For each module, there is a link to its meta server page. Here is a screenshot:

Completed client side authentication

Working on the client-side authentication code

First frontend steps

Implemented read-only anonymous access to resources and improved error handling

Implemented resource management

• The Resource interface simply states that a class implementing it is a resource and must provide a name.
• Then there are three main families of resources: WritableResource, ReadableResource and EventEmittingResource. As you can easily guess by the names, they define the methods that a resource must implement if it accepts data to be written to it, if it can provide data “on-demand” and if it emits events. The latest one is a class instead of an interface, because it implements internally the logic to register observers and notifying them.
• All the three kinds of resources are typed, i.e. a ReadableResource has a type parameter which determines the class of the objects which are obtained when reading this resource. WritableResource also has a method to get the Class object representing the type, which is necessary to deserialize client messages.
• The clients are identified by the same EntityRef objects which identify them in the engine.
• While EventEmittingResource and ObservableReadableResource look very similar, there are key differences between them. An EventEmittingResource produces data only by raising events, and can’t be queried directly; the ObservableReadableResource class is an addition to a ReadableResource (which can be read “on-demand”) that allows the subclasses to notify clients when the data has changed. At the data transport level, the server sends to WebSocket clients both the events generated by EventEmittingResources and the updates from ObservableReadableResources; for HTTP clients, the server can’t directly send data to clients, so events are saved in a queue that the client can query by periodically sending a GET request to a certain endpoint, while there is no “update queue” for ObservableReadableResources (it wouldn’t make sense to store, for example, all the changes to the number of connected users, which could be an ObservableReadableResource<Integer>); the client can simply send GET requests to periodically read the resource of interest to always obtain the latest value.
• (Not shown in the diagram) There is then a ResourceManager class which works as a registry of the instances of the resources and provides methods to access them according to their capabilities. The ResourceManager is used by the JsonSession class - which in turn is used by the classes which implement the WebSocket and HTTP protocols - to query or perform actions on the resources when requested by clients, or when a resource sends a notification.
• At the moment I implemented a ConsoleResource; clients can write strings (commands to be executed) to it and receive MessageEvents for each message that the server writes to the console.
• The simple WebSocket client has also been updated to keep compatibility with the backend changes; now, it also allows to send messages to query resources and it shows the received messages/events. Here is a screenshot showing the execution of a console command (the help command). There are still some character encoding issues - I guess the problem is in the serialization of MessageEvents - which will be fixed later.

A couple of words about HTTP and WebSocket

• The idea, as I mentioned in the GSoC proposal, is that the scenario where an user is connected to a Terasology server through a web application (which will be developed during the next weeks/months) is potentially an higly interactive environment suited for “push”-style notifications from the server to the client. For example when a new chat message (perhaps sent by another user connected through a regular Terasology client) is received by the server, the server must “send” (quotes since it can look unfamiliar if you are used to the “traditional” style of a web server only answering requests sent by clients) it to the client connected through the web interface.
• Until some years ago, this would have been possible only by having the client to periodically poll the server for updates. Nowadays, however, all the major browsers support WebSocket, a protocol much more suited for real-time bidirectional communication than regular HTTP.
• Providing only a WebSocket-based interface to the server, however, would be probably a little restrictive. One of the goals of the project is to provide an Application Programming Interface to allow arbitrary software to interact with a Terasology server (the other one is to provide a browser-based user friendly interface to it as described above); as a very simple example, one may want to retrieve the number of connected players from a shell script. Doing something like this via WebSocket would be cumbersome, if possible at all; you’d have to use a command line WebSocket client like wscat and in some way configure it to send a certain message to ask the server the required piece of information and wait for the server to answer. A much better way to accomplish a goal like this would be to have HTTP endpoints exposed by the server and querying them with curl.
• So, in the optimal setup, the server should expose both a WebSocket and a standard HTTP interface. In my understanding, this can be accomplished quite well by abstracting, in the server architecture, the accessible resources from the communication protocol. The JsonSession class handles the object serialization/deserialization with the JSON data format, but is not aware of the transfer protocol; other classes, WsHandler for WebSocket and HttpAPIServlet - added yesterday - for HTTP, work as a bridge between the transfer protocol and method calls to JsonSession instances. At the moment only the authentication handshake is handled, but this structure is designed to provide access to any resource, too, regardless of the used protocol. A request from the client to the server will result in a method of JsonSession being called and the returned object to be sent as the response, while an event generated by a resource would be notified by the JsonSession - using the Observer Pattern - to the appropriate handler and sent to the client by directly sending a message (if WebSocket) or by enqueuing it in a buffer that the user can read by requesting a certain endpoint (if HTTP).
• A diagram can probably express the concept better. Maybe this is not the best example since sending a chat message is an action that doesn’t need to be answered with data, but I think it still shows the main idea for handling requests from both the protocols:

• I implemented the HTTP interface to the authentication system. When the client initiates the handshake, a session token is generated and sent to the client; all the further requests must use that token in an HTTP header so that the server can verify the client’s identity. Code is in the HttpAPIServlet class.
• Then, I made a small proof-of-concept Javascript client which authenticates with a server using WebSocket. This is mainly to show that it’s possible to do the cryptography part (signing the verification data) in a browser, with the help of a library like jsrsasign. A problem I encountered is that it’s not simple to deserialize a “normal” configuration file as it’s saved by a Terasology client because of the big integers used in certificates. Looks like it’s not possible with the vanilla JSON.parse() but it’s necessary to use external libraries like json-bigint, but this should not be much of a problem since the real front-end client will most likely use browserify to manage browser-side libraries and thus easily support NPM modules such as json-bigint. For now, this very simple client only accepts a configuartion where the BigIntegers are serialized as the base64 of their binary representation, as in the client identity storage service. I posted the code (single file) as a gist, and here is a screenshot:

Completed authentication handler

• Added an HeadlessClient class, which extends the AbstractClient class defined in the engine and has the puropse to represent clients connected via the web-based API; it is instantiated as soon as the authentication is completed. By using certain helper methods already present in AbstractClient, HeadlessClient will take care of registering these clients with the engine’s entity manager. This is necessary to allow the clients to actually interact with the engine, for example to be notified of events, like chat or console messages.
• Improved the previously written code to manage the authentication handshake to send a server verification signature (a piece of data already known by both the client and the server signed with the server private certificate) upon succesfull authentication of the client, to allow the client to optionally verify the server’s identity using the server public certificate and optionally disconnect if the verification fails. I added this since the same action is made during the authentication of real clients.

Progress on client authentication via WebSocket

• The authentication scheme used for the web-based (or, more generically, external) clients is similar to the one already used to authenticate regular Terasology clients on a Terasology engine, and it’s based on signing the initial handshake messages with asymmetric cryptography; the appropriate public and private keys are stored in each peer’s (client or server) configuration file.
• Summing up, the new code contains a class that can be used to perform an authentication handshake, like the one implemented in the engine to handle connections from regular game clients, but not tied to a specific transfer protocol (TCP via Netty in the engine) or a data serialization format (Protobuf in the engine codebase).
• This class is then used by code which encapsulates a client-server communication based on the JSON data format; at the moment, this supports the authentication handshake (delegated, with proper serialization and deserialization, to the previously presented class) and will later be extended to manage requests and notifications to allow the clients to interact with the engine once authenticated.
• The final layer implements the communication protocol. At the moment, a simple WebSocket-based interface has been set up; it (sorry if this sounds repetitive) handles, at the moment, the authentication handshake, will be improved to handle requests (in the most general sense of the term, they could be status queries as well as action requests) from clients about the running engine and to send “live” server-to-client updates.

Connect FacadeServer to the Terasology engine

First days report

• The server code
• The pull request containing client code