- 23 July
Various small improvements to both client and serverSince the latest blog update I worked both on the server and on the client. Here is a summary of the code written during this week.
- It’s now possible to rename existing savegames.
- The class which provides information about available modules when creating a new game, AvailableModulesResource, is no longer a proxy for the meta server but shows only the modules installed on the server and also provides information about the available world generators.
- All the server-side code described in this an the previous blog updates is on a new pull request.
- There is a new screen with an animation which is shown when the client is trying to connect to the server, and if the connection fails the dialog which prompts for a server to connect to is shown again; this prevents the user to interact with carious controls before the connection is estabilished which could bring the user interface in an inconsistent state.
- It’s now possible to save addresses of favorite servers in the browser or device to avoid having to type them every time.
- In the dialog to create a new game, the world generator can now be chosen from a drop-down list (you no longer need to manually type the name). Also, the module list now only shows only the modules installed on the server; a separate panel to install/uninstall modules will be added later.
- Some parts have been refactored to improve the codebase (various TODOs have been removed) and the support for Android. This required an update of the ReactXP library to it’s latest version, which also introduced a small visual bug with text fields borders; I reported it and it should be fixed soon.
- Finally, I did some steps to allow authentication via an identity storage service, the feature is not available yet but it required to manually build two additional ReactXP controls.
- 16 July
Added support for various interactions with savegamesIn the previous update, I described how I implemented read-only access to the server’s savegame list; during this week, I added the possibility to interact with this resource. At the moment, it’s possible to perform these operations from the web interface or the web API:
- Start and stop games - if a game is running, starting another one will automatically stop the running game, while manually stopping a game will bring the engine to an idle state until another game is started
- Add a new game, by specifying the title, seed, list of modules and world generator to use
- Delete an existing game
- Generate a backup of a game - useful before doing “dangerous” operations like adding or removing modules from an existing savegame, which will be implemented soon
Here is a screenshot of the “New Game” dialog of the web frontend application:
As you may have noticed, the module dependencies are not resolved on the client side; in fact, for example, Core Gameplay depends on Terasology Engine, but the latter is not shown in the enabled modules list. The dependencies, at the momet, are resolved only on the server side when the user clicks OK; when the server builds the new game manifest, it adds all the necessary dependencies. A possible improvement could be to show the dependencies on the client side too, maybe together with a description for each module.
The available modules list (the right column in the new game dialog) is obtained from the meta server API; however, this can’t be done directly because the meta server at the moment does not support CORS so the game server is being used as a proxy (see the AvailableModulesResource class). To improve this hack that I put together to circumvent the no-CORS problem, there are two directions I can think of: the simplest could be to send a pull request to the meta server repository which adds a wildcard Access-Control-Allow-Origin header to the public API endpoints, thus making them accessible to any browser (in my understanding this shouldn’t be a security problem since it’s just public data); alternatively, however, AvailableModulesResource could be improved to not be a mere proxy but instead provide additional useful information, like the current availability of the module on the server.
At the moment, I’m working on implementing additional features to allow modification of existing savegames, more specifically renaming them and changing the module list. Another quite important feature on the to-do list is allowing an administrator to login with a password - set when the server is started - at the first run of a server; in fact, the addition of the Idle engine state has decoupled the existence of a running server from the existence of a savegame generated with the default options. In other words, it’s possible to start the server without generating the default game (using the -dontStartDefault command line switch) to manually configure the games using the web interface, but currently there is no way to actually access the web interface when a new server is started for the first time with that CLI argument, because the only way to login to the web interface is by using client identity certificates, which are generated when connecting to the server through a regular Terasology client - which is not possible when the server is not running any game yet.
- 9 July
Implemented read-only access to savegames and modulesAt the end of the previous update, I mentioned that the next goal was to allow to work with the server configuration from the API/web interface. Here are some relevant points that must be taken in consideration to implement this feature:
- A key feature of Terasology is modularity, so the web interface should allow server administrators to add/remove or enable/disable the gameplay modules.
- A Terasology instance can manage multiple savegames (though only one is running at a given moment), each of them with potentially different sets of modules enabled. At the moment, this is mostly used in the “client” mode, where a user can have various single-player savegames; the headless server implementation (see StateHeadlessSetup.java) creates a single savegame with a certain module list, which can be customized by editing the configuration file.
- To enable or disable modules on a certain savegame, that game must not be running. Also, usually the module list is set when the game is created and then left unchanged; I still have to investigate whether it’s possible to add or remove modules on an existing savegame, it probably depends on the module.
- The permission system is internal to each savegame, i.e. a client identified by a certain client ID can have different permissions in different games and there isn’t a “global” permission system.
These considerations led me to the following conclusions:
- Two different resources should be added, one to manage the various savegames and their modules (I’m working on this one at the moment) and one to manage “global” server settings (such as the network port).
- A new engine state which is not running any game is required to allow the server administrator(s) to configure modules for each game. When the engine is in this state, the administrator can create one or more games in the web interface and choose one to run.
- To determine whether a client is a server administrator, a list of the client IDs allowed to perform these restricted task could be saved in a file in the server data directory.
- A command line switch can be used to tell the server if, at first startup, it should generate a game with the default modules or wait for a connection from the web interface to let the administrator manually configure the game, then start it. In a similar way, another command line argument should allow to select which game sould be started automatically if there are multiple ones.
- To allow the administrator to configure a game on the first startup, an additional authentication method needs to be added. In fact, on the first startup, there are no client identities stored in the server configuration file, because they are generated when an user connects through a normal client (not the web application). A simple solution could be to specify a password on the command line
or as an environment variable when starting the server instance for the first time, which could then be entered in the web interface.
At the moment, I implemented read-only access to the server status and the game list with module information (both the server-side, on the new games-resource branch, and the client-side, in the frontend repository). For each module, there is a link to its meta server page. Here is a screenshot:
The next goal is to provide write access (creating new games, selecting modules, stopping a game and running another one), restricted to server administrators, as described above.
- 6 July
Completed client side authenticationI’m writing this short DevBlog update to confirm that I finally managed to get the client side authentication code to work. All the relevant code is in the latest commits pushed to the frontend repository. To achieve this goal I used the jsrsasign and jsbn libraries to perform the RSA signature of the handshake message expected by the authentication protocol. At the moment, the public and private certificates can only be supplied by pasting the whole game client configuration file in the web interface; in the upcoming weeks I plan to add support to authentication based on identity storage service and maybe an option to store the certificates in the browser’s localStorage to avoid having to authenticate every time the page is loaded.
Here is a screenshot of the authentication UI:
The working authentication code makes it possible to execute console commands; in the following screenshot, for example, the
helpcommand is used to obtain a list of the available commands, then the
saycommand is used to send a couple of chat messages. As you can see, the text colors are supported too.
As you may have noticed from these two screenshots, the web application is now being deployed to GitHub Pages, so you can always run the latest working version (the Travis-CI service is set to automatically deploy the working builds) by visiting https://gianluca-nitti.github.io/FacadeServer-frontend/.
At the moment, I’m working on adding access to server’s configuration to FacadeServer and to the client.
- 2 July
By the way, I was notified that I passed the first GSoC evaluation. I’m very happy of this and I thank my mentors, the MovingBlocks community in general and Google for supporting this and other projects and for the feedback they are providing.
- 26 June
Here is a screenshot of the actual state of the frontend application, in the web version running in a desktop browser: As you can see, I’m going to organize the interface in tabs which can be swithed using the menu on the left (which I should probably make a popup when the viewport is under a certain width, to avoid taking too much space on mobile devices in portrait orientation), to access different resources such as console, configuration, etc; at the moment, there are a console tab and a home tab that shows a list of the connected players. All the updates are received in real-time thanks to WebSocket. At the moment, the client connects in anonymous mode and thus is read-only (i.e. it’s not possible to run console commands), and adding the code to perform authentication - by pasting the JSON client configuration or logging in to an identity storage service account - is my next goal. Should be quite simple since it will mostly consist in tidying up the code I already written for testing and integrating it in the frontend.
- 22 June
Implemented read-only anonymous access to resources and improved error handlingAs said at the end of the previous post, in the last few days I wrote the code to allow unauthenticated clients to get read-only access to some resources. So, for example, a list of the currently online players on the server can be obtained by simply sending an HTTP GET request to
http://server-address/http/resources/onlinePlayersand you’ll get a JSON array containing the player list, without having to perform the authentication process which would involve identifying with a client identity certificate with a few more requests which, if the identity is correctly verified, would provide the client with the token to perform authenticated requests. Obviously, this is still necessary to perform actions which have effects on the server, for example running console commands, but since there are informations (like, as said, connected players, server metadata, etc) which can be made publicly available I thought that allowing anonymous access to them was handy.
The example above uses HTTP, but access via WebSocket is available as well. In this case, once a WebSocket client is connected, it will immediately start receiving “public” events (like notifications when a player connects) and is allowed to perform read requests to resources. Then, it can perform authentication at any time to obtain write access and subscribe to “private” notifications (like chat messages sent - or “whispered” - only to the player identified by the client identity that the WebSocket client used to authenticate).
A limitation is that event notifications for anonymous clients (for example public chat messages) are available via WebSocket but not via HTTP. This is because for HTTP, events are put in a per-client queue which is emptied when the client requests a certain endpoint to receive the pending events which happened since the previous request; this is possible for authenticated clients which are identified by a secret session token and thus are the only ones that can empty their queue, but not for anonymous clients. For example, we can consider the following situation: there are two anonymous clients,
B, which query the event endpoint every 5 seconds.
Astarts querying it and is notified of an event
Xand plans to repeat the request after 5 seconds; a second later an event
Brequests the event queue:
Bis notified of
Y, but also empties the queue, which means that
Y. Thus, it wouldn’t make sense to have a publicly shared event queue managed in this way because not all clients would be notified of all events. If this feature is necessary or useful, I could later consider alternative ways of implementing it, for example keeping a fixed number of recent events in the public queue and avoid emptying it on each request.
All the code is on the
resourcesbranch, which is now the Pull Request #3 (by the way, PR #2 has been recently merged). At the moment, my focus is to start the development a browser-based frontend with basic functionality (like showing connected players list, allowing authentication and console usage).
- 18 June
Implemented resource managementSince the latest update I mainly worked on the code to provide access to certain resources to WebSocket and HTTP clients; the code is on the resources branch. Here is a class diagram showing the architecture of the interfaces and classes which represent resources, together with the Console resource (the only one available at the moment): Some notes:
- The Resource interface simply states that a class implementing it is a resource and must provide a name.
- Then there are three main families of resources: WritableResource, ReadableResource and EventEmittingResource. As you can easily guess by the names, they define the methods that a resource must implement if it accepts data to be written to it, if it can provide data “on-demand” and if it emits events. The latest one is a class instead of an interface, because it implements internally the logic to register observers and notifying them.
- All the three kinds of resources are typed, i.e. a ReadableResource has a type parameter which determines the class of the objects which are obtained when reading this resource. WritableResource also has a method to get the Class object representing the type, which is necessary to deserialize client messages.
- The clients are identified by the same EntityRef objects which identify them in the engine.
- While EventEmittingResource and ObservableReadableResource look very similar, there are key differences between them. An EventEmittingResource produces data only by raising events, and can’t be queried directly; the ObservableReadableResource class is an addition to a ReadableResource (which can be read “on-demand”) that allows the subclasses to notify clients when the data has changed. At the data transport level, the server sends to WebSocket clients both the events generated by EventEmittingResources and the updates from ObservableReadableResources; for HTTP clients, the server can’t directly send data to clients, so events are saved in a queue that the client can query by periodically sending a GET request to a certain endpoint, while there is no “update queue” for ObservableReadableResources (it wouldn’t make sense to store, for example, all the changes to the number of connected users, which could be an ObservableReadableResource<Integer>); the client can simply send GET requests to periodically read the resource of interest to always obtain the latest value.
- (Not shown in the diagram) There is then a ResourceManager class which works as a registry of the instances of the resources and provides methods to access them according to their capabilities. The ResourceManager is used by the JsonSession class - which in turn is used by the classes which implement the WebSocket and HTTP protocols - to query or perform actions on the resources when requested by clients, or when a resource sends a notification.
- At the moment I implemented a ConsoleResource; clients can write strings (commands to be executed) to it and receive MessageEvents for each message that the server writes to the console.
- The simple WebSocket client has also been updated to keep compatibility with the backend changes; now, it also allows to send messages to query resources and it shows the received messages/events. Here is a screenshot showing the execution of a console command (the help command). There are still some character encoding issues - I guess the problem is in the serialization of MessageEvents - which will be fixed later.
- 13 June
A couple of words about HTTP and WebSocketUntil now, I talked about WebSocket, HTTP and used the generic term “web API” without going into much detail; probably, it’s time (maybe a bit late) to clearly explain on this site what king of interface will be available to externally access a Terasology server running inside FacadeServer.
- The idea, as I mentioned in the GSoC proposal, is that the scenario where an user is connected to a Terasology server through a web application (which will be developed during the next weeks/months) is potentially an higly interactive environment suited for “push”-style notifications from the server to the client. For example when a new chat message (perhaps sent by another user connected through a regular Terasology client) is received by the server, the server must “send” (quotes since it can look unfamiliar if you are used to the “traditional” style of a web server only answering requests sent by clients) it to the client connected through the web interface.
- Until some years ago, this would have been possible only by having the client to periodically poll the server for updates. Nowadays, however, all the major browsers support WebSocket, a protocol much more suited for real-time bidirectional communication than regular HTTP.
- Providing only a WebSocket-based interface to the server, however, would be probably a little restrictive. One of the goals of the project is to provide an Application Programming Interface to allow arbitrary software to interact with a Terasology server (the other one is to provide a browser-based user friendly interface to it as described above); as a very simple example, one may want to retrieve the number of connected players from a shell script. Doing something like this via WebSocket would be cumbersome, if possible at all; you’d have to use a command line WebSocket client like wscat and in some way configure it to send a certain message to ask the server the required piece of information and wait for the server to answer. A much better way to accomplish a goal like this would be to have HTTP endpoints exposed by the server and querying them with curl.
- So, in the optimal setup, the server should expose both a WebSocket and a standard HTTP interface. In my understanding, this can be accomplished quite well by abstracting, in the server architecture, the accessible resources from the communication protocol. The JsonSession class handles the object serialization/deserialization with the JSON data format, but is not aware of the transfer protocol; other classes, WsHandler for WebSocket and HttpAPIServlet - added yesterday - for HTTP, work as a bridge between the transfer protocol and method calls to JsonSession instances. At the moment only the authentication handshake is handled, but this structure is designed to provide access to any resource, too, regardless of the used protocol. A request from the client to the server will result in a method of JsonSession being called and the returned object to be sent as the response, while an event generated by a resource would be notified by the JsonSession - using the Observer Pattern - to the appropriate handler and sent to the client by directly sending a message (if WebSocket) or by enqueuing it in a buffer that the user can read by requesting a certain endpoint (if HTTP).
- A diagram can probably express the concept better. Maybe this is not the best example since sending a chat message is an action that doesn’t need to be answered with data, but I think it still shows the main idea for handling requests from both the protocols:
- I implemented the HTTP interface to the authentication system. When the client initiates the handshake, a session token is generated and sent to the client; all the further requests must use that token in an HTTP header so that the server can verify the client’s identity. Code is in the HttpAPIServlet class.
- 11 June
Completed authentication handlerI think that the authentication handling code can now be considered completed. Since the last update I:
- Added an HeadlessClient class, which extends the AbstractClient class defined in the engine and has the puropse to represent clients connected via the web-based API; it is instantiated as soon as the authentication is completed. By using certain helper methods already present in AbstractClient, HeadlessClient will take care of registering these clients with the engine’s entity manager. This is necessary to allow the clients to actually interact with the engine, for example to be notified of events, like chat or console messages.
- Improved the previously written code to manage the authentication handshake to send a server verification signature (a piece of data already known by both the client and the server signed with the server private certificate) upon succesfull authentication of the client, to allow the client to optionally verify the server’s identity using the server public certificate and optionally disconnect if the verification fails. I added this since the same action is made during the authentication of real clients.
- 8 June
Progress on client authentication via WebSocketSince the latest update I’ve been working mainly on this branch.
- The authentication scheme used for the web-based (or, more generically, external) clients is similar to the one already used to authenticate regular Terasology clients on a Terasology engine, and it’s based on signing the initial handshake messages with asymmetric cryptography; the appropriate public and private keys are stored in each peer’s (client or server) configuration file.
- Summing up, the new code contains a class that can be used to perform an authentication handshake, like the one implemented in the engine to handle connections from regular game clients, but not tied to a specific transfer protocol (TCP via Netty in the engine) or a data serialization format (Protobuf in the engine codebase).
- This class is then used by code which encapsulates a client-server communication based on the JSON data format; at the moment, this supports the authentication handshake (delegated, with proper serialization and deserialization, to the previously presented class) and will later be extended to manage requests and notifications to allow the clients to interact with the engine once authenticated.
- The final layer implements the communication protocol. At the moment, a simple WebSocket-based interface has been set up; it (sorry if this sounds repetitive) handles, at the moment, the authentication handshake, will be improved to handle requests (in the most general sense of the term, they could be status queries as well as action requests) from clients about the running engine and to send “live” server-to-client updates.
- 5 June
Connect FacadeServer to the Terasology engineToday I opened the first pull request on FacadeServer, the main repository where my GSoC code is going to be stored. The proposed code connects the codebase, which is actually a Jetty-based webserver, with the main Terasology engine, and runs it in headless (or server) mode (on the main thread), while, at the seme moment, it continues to listen and answer for HTTP and WebSocket connections on the threads spawned by Jetty.
A more detailed explanation of the work done is available in the pull request’s description, at the link above.
The next goal is to set up the authentication protocol and code to allow the server to identify clients connecting via HTTP/WebSocket according to the game server configuration.
- 2 June
First days reportImproved and completed the Terasology identity storage service, a sub-project I started earlier which consists of a web service to store the Terasology client identities and synchronize them across clients. This is useful if you have multiple computers, and will make the authentication for the web interface to the Terasology servers which I’ll build as part of the GSoC more user-friendly.