Pubserver software

General description

Purpose of this document

This document describes the software used to drive pubserver, the public set of Freeciv servers kept running at pubserver.freeciv.org.

This document is supposed to reflect as closely as possible the actual situation, and concerns only the original system of pubserver software developed mostly by Reinier Post and Paul Zastoupil.

Specifically, it does not describe publite and other initiatives to replace parts of this system with something else, since they are not ready to take over production just yet.

Please do not use this document to describe wishes or plans regarding the pubserver software or discuss its design; the Pubserver Internals document exists for that purpose.

See also pubserver system administration.

Purpose of the pubserver software

The specific purpose of pubserver is to provide a public Freeciv service: it runs a battery of Freeciv servers that users can connect to from all over the Internet with their Freeciv clients to play Freeciv games. All this is installed under the freeciv account's home directory.

The software and website are under CVS control, with its repository on cvs.freeciv.org; it has not been made publically available.

Portability

The software is quite portable; the build system used to be cross-tested on a Solaris host (svis02.win.tue.nl, to be exact). It may be an idea to start crosstesting it on Cygwin, this shouldn't pose much of a problem.

One file, share/Makefile, depends on GNU make.

Some of the auxiliary tools outside the build system (i.e. not called by civservers) are Linux specific (e.g. depend on the Linux /proc filesystem).

Filesystem layout

One way to look at the software is to see where it lives on the file system.

Everything is owned by the freeciv user and can be found under ~freeciv (some things are symlinked there from somewhere else). (Note that it would be safer to have separate users for separate parts.)

The relevant directory trees are:

~freeciv/bin

command line utilities for maintenance and monitoring, the build system, and post-game reporting (mostly scripts written in /bin/sh and Perl)

~freeciv/share

auxiliaries for these executables (e.g. an important Makefile)

~freeciv/.freeciv/code: Freeciv civserver executables with supporting data, at various stages of compilation and installation (the process is controlled from share/Makefile)

~freeciv/.freeciv/games

the working directories of all running games ~freeciv/.freeciv/*

config files, abuse reports, etcetera

~freeciv/htdocs/games

the game reports on completed games

~freeciv/htdocs/*

the rest of the website, with dynamic pages using PHP

This does not completely organize everything by its function; it really should. For example, all the different kinds of command line utilities live together in ~freeciv/bin, together with a couple of throwaway and dysfunctional scripts, which is quite inconvenient.

Post-game data are kept partly as files on disk, partly in a MySQL database on www.freeciv.org.

The pubserver software, organized by function

Overview

In all, the pubserver software consists of over 40 scripts; they will now be described organized by their function within the whole.

The most logical functional decomposition seems the following:

the build system

a system of command line tools that assembles and runs a battery of Freeciv servers

post-game reporting

a system of command line tools that archives game results and creates (not so) pretty reports for them

cleanup

command line tools to erase working directories that are no longer needed

monitoring

additional command line tools for the pubserver admin to see what is going on

website

displays the pubserver status and the post-game reports to end users

The build system

The software to operate the Freeciv servers consists of a couple of scripts that call each other. Here is most of the call graph, showing only calls to homemade utility scripts - calls to external utilities (make, GET, patch, gnuplot, gifsicle, etc. etc.) are not included.

bin/civservers
|
+-+> share/Makefile
  |
  +-+> fetch
  |
  +-+> patch
  | |
  | +-+> bin/cvsdiff2udiff
  | |
  | +-+> bin/p0p1
  |
  +-+> config
  |
  +-+> compile
  |
  +-+> diff-patch, diff-compile
  |
  +-+> install
  | 
  +-+> start, restart
  | |
  | +-+> bin/civserver-forever
  | | |
  | . +-+> bin/cs
  | .   |
  | .   +-+> bin/newsubdir
  |       |
  |       +-+> .freeciv/code/(...)/civserver
  |
  +-+> stop
  | |
  | +-+> bin/civserver-kill
  |
  +-+> wipe-fetch, wipe-patch, wipe-compile, wipe-install
  |
  +-+> fetched-in, patched-in, compiled-in, installed-in

    .
    .
    +-+> picture-civgame
      |
      +-+> -d  (database update)
      | |
      | +-+> bin/serversettings
      | |
      | +-+gt; bin/doranking
      | |
      | +-+gt; /home/hirisov/script/doranking, /home/hirisov/script/tourny
      | |
      | +-+gt; bin/xmlgamelogparse
      |
      +-+> -a  (analyse core dump)
      | |
      | +-+> bin/civserver-core-backtrace
      |
      +-+> -c  (copy game data)
      |
      +-+> -r  (generate animated replay map)
      | |
      | +-+> bin/sortsav
      | |
      | +-+> bin/ppm2gif
      | |
      | +-+> bin/fcivppsav
      | |
      | +-+> bin/fcivppint
      | |
      | +-+> bin/fcivmaprace
      | |
      | +-+> bin/fcivmapplayers
      |
      +-+> -p  (generate score plots)
        |
        +-+> bin/savs2time
        |
        +-+> bin/civscore2gnuplot
        |
        +-+> bin/colorize-gnumultiplot-ps

Top-down:

bin/civservers

The purpose of this script is twofold:

• it stores the pubserver configuration details, listing which server configuration runs on which port
• it acts as the general control shell

It may not seem logical to let a script be its own configuration file, but it keeps things in one place and provides the flexibility required.

Since civservers is its own configuration file, it contains a minimal amount of code: all actual functionality is delegated to share/Makefile.

The script can be called without arguments to perform whatever is required to get everything going, or with specific arguments to perform specific subfunctions.

The calling syntax is /etc/init.d idiom (civservers start, etc.) extended to support all the various subfunctions (civservers patch, civservers install, etc.). Subfunctions depend on each other (e.g. you can't compile without having applied the indicated patches, you can't apply patches without having fetched the indicated code base). This is why civservers is just a wrapper around share/Makefile and its arguments are the Makefile targets.

There are special targets of the form wipe-* to forcibly remove the results of a particular step; e.g. wipe-patch erases the results of patch, so re-patching can be forced.

There are two ways of making this script display its actions without executing them:

• civservers -n [action] will display the make command that would be invoked, without actually doing so
• civservers <action> -n will execute the make command with the -n option, causing make to display its actions without executing them

Sorely missing is an option to execute civservers on just a specific port or range of ports; however, the -n option can be used to get around this.

Called by cron, but often used on the command line.

It is important to be aware that this script runs automatically from cron. Certain configurations that are easy to choose by mistake (e.g. telling it to recompile from current CVS on every run) will make the host spin out of resources.

share/Makefile

Automatically creates and deploys a civserver from specifications supplied as command line options and/or environment variables.

Designed to support running different Freeciv servers, compiled from different codebases, with possibly different patch sets, alongside each other on different ports. This feature is needed since pubserver always needs to support at least two different release versions, and a varying range of pubserver-specific patches; it is also used to experiment with certain experimental server-only features, and sometimes, to merge in nonstandard rulesets (they can be repackaged as patches).

This is implemented by encoding the arguments relevant to a stage into the names of the working directories used. This is done in a way that maximizes reuse: e.g. two versions based on the same CVS snapshot will only cause the CVS codebase to be checked out once, even when different sets of patches are subsequently applied to it, while a different CVS timestamp will cause a different cvs checkout into a different directory. The disadvantage is that the directory names become very long and unreadable. Also, the implementation is quite ugly.

The following Makefile targets represent the consecutive stages of the build process:

fetch

fetches a source code tree from a tarball on the web or from CVS (SVN support will be added soon)

patches

assembles the list of patches to apply (should fetch them from rt.freeciv.org; there is some code to support this, but the change was never finished)

patch

patches the source code with the list of patches (if any) (presently found in ~freeciv/patches, should be obtained from RT)

compile

configures and compiles the resulting source code tree

install

installs the compiled tree into a different directory

start

deploys an installed server on one or more TCP ports

Each stage except the last stores its results in a different subdirectory of ~freeciv/code. The last stage calls bin/civserver-forever to do the real work.

Other operations (i.e. symbolic Makefile targets) include:

stop

forcibly kills a running server

restart

stops and restarts a server, unless any user is connected to it

diff-patched, diff-compiled

creates a patch from a (possibly edited) codebase - very useful when you are updating existing patches to work with the present codebase

wipe-fetch, wipe-patch, wipe-compile, wipe-install

cleans out the result from the respective build stage

fetched-in, patched-in, compiled-in, installed-in

prints the paths of the working directories used in the respective build steps so you can cd to them (they are not human readable)

Most steps can be configured in various ways with Makefile variables and environment variables. This Makefile is not intended to be invoked directly; the organized way to invoke and configure it it is through the bin/civservers wrapper.

Although many functions are performed by command line tools in bin/*, quite a large amount of code has built up directly within this Makefile, which makes it hard to read and maintain. The individual steps (targets) should be factored out to scripts.

Called from bin/civservers. Unless you are digging all the way down to the bottom, please don't call this Makefile directly. A useful practical hack is to generate invocations with civservers -n, copy a result to the shell prompt, edit it manually, then invoke it.

bin/cvsdiff2udiff

Modifies patches (mainly those from CVS) to a unified diff format.

Called by civservers patch, but useful on its own.

bin/p0p1

Looks at a Freeciv patch to determine whether to patch it with -p0 or -p1.

Called by civservers patch, but useful on its own.

bin/civserver-forever

Keeps a civserver running in an endless loop, by restarting it when it quits. Server output is saved to civserver.out, which is important in investigating abuse reports.

The present version uses civserver's -quitidle flag to make the server quits when no users are present. (Earlier versions depended on parsing the server output to determine this condition and sending a /quit response on input when it was met.)

The present version also initiates post-game reporting, which can alternatively be started independently from cron.

Called by civservers start and civservers restart, but useful on its own.

bin/cs

Starts a civserver in a newly created subdirectory under ~freeciv/.freeciv/games. This guarantees that two different civservers do not overwrite each other's output data, such as savegames and game logs. The directory name is a counter; many scripts use it to identify the game, e.g. on the website.

Called by bin/civserver-forever, but useful on its own.

bin/newsubdir

An auxiliary script to create a new subdirectory.

Called by bin/cs, but useful on its own.

bin/onlyone

Prepend this to your command line to make sure the invoked command is only executed when it isn't executing already.

Very useful on long-term or costly commands (such as picture-civgame), especially when invoked from cron.

The method to determine whether the command is already running is a trial-by-error heuristic and by no means foolproof.

Called by various other scripts, but useful on its own.

bin/impatiently

Prepend this to your command line to make sure the invoked command is terminated unless it starts to produce output within a given time interval.

Very useful on long-term commands or commands that may block (such as civserver-ping)

The method to properly administer the timeout and exact the kill is a trial-by-error heuristic and by no means foolproof.

Called by various other scripts, but useful on its own.

bin/civserver-kill

Kills the civserver process(es) running on the port(s) supplied on the command line.

Called by civservers stop, but useful on its own.

Post-game archiving and reporting

Another set of scripts has the purpose of condensing the data dumped on disk by an ended civserver to a manageable quantity and generating Web-accessible overviews.

bin/picture-civgames

This script can be called to do post-game processing on existing game directories. It simply calls bin/picture-civgame on the specified game directories.

Designed to be called by cron, but useful on its own.

bin/picture-civgame

This script performs all post-game processing on a game. It serves four purposes:

• running reporting software on the game data to display (not so) pretty statistics and graphs to end users
• condensing and archiving all relevant game data into a subdirectory on the webserver, so the original working directory can be cleaned out;
• recording game results into the game database (including a computation of ranking information)
• alerting Freeciv admins in case of problems

As you can tell from its name, the script started out doing only the first part; it created an animated replay map and some gnuplot gaphs and saved them to a directory on the website for viewing with htdocs/viewgame.phtml.

Later, the information saved to that directory was extended in two ways:

• game information was added that doesn't show up to end users (e.g. the civserver output, to investigate abuse, and a coredump, if the server crashed)
• intermediate information was added so the graphing can be re-run (so we can improve the graphing part and rerun it on old games)

Furthermore, game information was saved to a MySQL database; we now depend on it for authentication and ranking. Two different ranking systems are invoked, in fact.

Finally, when the server is found to have crashed, a core dump backtrace is emailed to the Freeciv admins.

Like bin/civservers, the script can be called without arguments, in which case it will do everything, or with arguments, to execute a particular subfunction. Care was taken to make all execution re-entrant: the script and its subfunctions can be rerun without leading to invalid results. The exception is the database update and ranking computation, for which I (rp) don't know how to make them reentrant; see the descripton of bin/doranking for details.

Takes care not to execute on games that are still running: the results would generally be invalid and should not yet be available anyway.

By default, will skip subfunctions that have been executed before; does not employ a Makefile to implement this. Re-execution can be forced with the -f option.

Called by bin/picture-civgames and bin/civserver-forever, but useful on its own.

bin/serversettings

Inserts game results into the game database.

Called by bin/picture-civgame. Ought to be reentrant.

bin/doranking

Computes player ranking updates based on the game results, and inserts them into the game database.

Called by bin/picture-civgame. Ought to be reentrant.

The Pubserver Internals document discusses details of the ranking system.

/home/hirisov/script/doranking, /home/hirisov/script/tourny

An improved ranking system designed and implemented by hirisov. Called by bin/picture-civgame.

Ought to be reentrant; since ranking depends not only on the present game's results, but also on the participants' rankings at the time the ranking is updated, this is not so easy to achieve. Ideally, the update should be history independent, and e.g. record a timestamp to indicate the moment of update; that way, both the update and the ranking computation become reentrant.

bin/xmlgamelogparse

Performs ranking based on a 2.1 (XML) gamelog.

Called by bin/picture-civgame, but useful on its own.

bin/civserver-core-backtrace

Runs gdb to create a backtrace of a civserver coredump (if present).

Called by bin/picture-civgame, but useful on its own.

bin/sortsav

Called by bin/picture-civgame, but useful on its own.

bin/ppm2gif

Not a homegrown script, but a standard compiled executable (from Jef Poskanzer's netpbm library) to convert images from PPM to GIF format.

Called by bin/picture-civgame.

bin/fcivppsav, bin/fcivppint, bin/fcivmaprace, bin/fcivmapplayers

Perl scripts that extract useful information from the series of savegames left by a game. Based on common code in share/perl.

Called by bin/picture-civgame.

There is a performance bottleneck here.

bin/savs2time

From a series of Freeciv savegames and their mtimes, computes actual time consumption and timeout information, and saves it in Freeciv scorelog format.

Called by bin/picture-civgame, but useful on its own.

bin/civscore2gnuplot

Takes information in Freeciv scorelog format and graphs it with gnuplot, to produce the game graphs shown in htdocs/viewgame.phtml.

Called by bin/picture-civgame, but useful on its own.

bin/colorize-gnumultiplot-ps

Recolorizes gnuplot graphs produced by bin/civsciore2gnuplot, since games may feature more players than the number of different colors gnuplot can emit.

Called by bin/picture-civgame.

~freeciv/share

Contains auxiliary files (mostly code) that aren't executable scripts:

share/Makefile

the heart of the autobuilding system for civservers

share/perl

a Perl library to parse with Freeciv game data

The Makefile is discussed elsewhere.

share/perl

A simple Perl library, written by Todd Goodman, is available for parsing Freeciv savegames: Freeciv::SavFiles. It is used in utility scripts called by picture-civgame.

Cleanup commands

Some are automatically invoked by cron.

bin/cleanup-gamedata

Removes working directories of Freeciv servers that are no longer running. Arguments specify criteria for removal.

Called from cron, but mostly useful on the command line (cron is careful not to remove too much).

bin/cleanup-unused-code

Removes the directory trees under ~freeciv/.freeciv/code that are no longer used by the present bin/civservers configuration.

Determining this is a little tricky, which is why it is not called from cron.

bin/civgame-clean

Reduces the number of savegames in a server's working directory, as if civserver had been invoked with the saveturns parameter supplied as argument.

Not used.

Monitoring commands

Some utilities support the pubserver admin logging on to pubserver to see what is going on:

bin/civserver-port-pid-dir

The primary way to survey what is running on pubserver. For each running server, lists its pubserver port, process id, and working directory. Within the working directory, files are to be found that show the civserver version and startup configuration.

It is important to note that the set of running servers typically does not correspond to the set configured in bin/civservers. A different configuration will only take effect on a port after the running civserver has been terminated, which will only happen when no more users are connected to it. Furthermore, installing new configurations may fail, e.g. due to errors in the patching or compiling stages.

bin/civserver-connections

Lists the running civservers with their port and the hostnames of the connected users.

bin/civserver-ping

Pings users connected to a given pubserver port. Useful in determining connectivity problems.

This was important in the pre-1.12.0 days, when poor connectivity from one user would routinely lock up the server for everybody else. Not much used today.

bin/civserver-status

Another way to list the status of running civservers.

bin/civserver-quit

Issues a /quit command to a running civserver. No longer works, because civserver-forever no longer supports supplying server command line input to the civservers it starts. (This feature was rarely used, but it allows the admin some nice emergency measures, and some other scripted behavior. For example, the metaserver infostring could be automatically set every 15 minutes.)

bin/freecivs

An old client side script. Lists the Freeciv servers shown on the Freeciv metaserver and optionally invokes Freeciv clients to connect to them.

Not used. Not updated to support the 2.x metaserver.

bin/free-ports-between

Finds ports on which no server is running.

Called by share/Makefile, but useful on its own.

bin/proc-net-tcp2host-port

Roughly, a non-civserver-specific version of bin/civserver-port-pid-dir.

bin/diskspace-check

Estimates whether enough disk space is available for running games.

Called by cron, but useful on its own.

bin/*

Scripts for which a description is still missing:

civcities
civparse
civpath2server
civpow
civyears2turns
cpfreeciv
doallranking
fix-civscore
forxcpu
gamelog2html
getnumbers
mkpatch
names
nrframes
report-civscore
sav2ppm
sav2score
save4patch
savplayers
savyear
tar.gz-civserver
testcolorcube
testxcpu

The pubserver website: ~freeciv/htdocs

The pubserver website displays information on past and present pubserver games. It combined preprocessed information with information from the games database and on-the-fly formatting, using PHP as the scripting language.

The main pages on the webserver:

htdocs/index.phtml

The main page; contains a table with news items that requires updating after user-visible changes. Its output is automatically copied to index.html for performance reasons.

htdocs/viewgame.phtml

Reports on an individual game. Most reporting data, such as the animated replay map and the game statistics histograms, are pregenerated, but this script is still quite slow. Some optimizatiuon has been done, e.g. in the generation of URLs for the previous and next buttons, but more is required.

The output doesn't look too good.

Filtered havigation (e.g. a button to go to the next game matching certain criteria) would also be good to support.

htdocs/query.phtml

Displays a table of games from the (MySQL) database. Supports sorting and selecting by specific columns, and crosslinks with viewgame.phtml, but the interface is quite primitive; more features and columns could be added, and the interface could probably be made easier to use.

htdocs/rankings.phtml

Displays a table of players from the database, by ranking. The ranking system, designed by Paul Zastoupil and Micha Riser, has known strengths and weaknesses. See also bin/dorankings.

htdocs/*

Other webpages: a gamelog prettyprinter (contributed by Sönke Ufen), a directory of IRC users, a (cyphor) message board, and more.

The pubserver data

An overview of the data employed by the pubserver software.

Data in the MySQL database

TO DO

Data on the file system

FAR FROM COMPLETE.

~freeciv/code

pubserver.freeciv.org always runs multiple Freeciv versions and always applies various special patches to the source code.

In January, 2003, for example, the following versions were running:

1.13.0

because Linux distributions were packaging it

1.14.0-beta2

with the "teams" patch (a popular feature)

1.14.0

the latest stable release, against which the "teams" patch would not work

current CVS

always available for playtesting, updated automatically

All have extra patches applied to fix bugs and add special behaviour. For instance, abuse in the past has led to a very simple patch that sets the minimum timeout to 40. Another patch adds a server command to add observer players to the game.

A civserver binary requires a matching data directory for rulesets, nations, etcetera. Therefore, every different variant is installed (with make install into a separate directory tree, and started from there.

These trees are under ~freeciv/.freeciv/code/installed. The other subdirectories of ~freeciv/.freeciv/code are working directories used during the compilation process. The trees are subdirectories of freeciv/code/installed.

~freeciv/games

The base directory for running civservers; the cs wrapper script creates a subdirectory when it starts a civserver.

We once lost a lot of data when the number of subdirectories grew to more than the Linux kernel could handle, but this was fixed with a kernel upgrade. No performance issues have been noticed, although something like an ls, due to the sorting it does, should obviously be avoided in scripts when possible.

~freeciv/*

Some other directories exist under ~freeciv; the incidents directory contains reports and logs of individual incidents, mostly cases of obnoxious civserver user behavior.

~freeciv/htdocs/games

The civserver reporting scripts take the data left by an ended game in ~freeciv/.freeciv/games and deposit their results under this directory. Most of the results are linked to from viewgame.phtml, the PHP script that displays an overview for an individual games; some are not; a few are password protected because they contain users' IP numbers.