======= Compressed maps files =======
When operating the [[:server]] , the [[:Bigworld]] map files in
[[environment_variables|CROSSFIRE_LIBDIR]]**/maps** can take up to 400 MB of space on the hard-disk.
Until version 1.60.0 and below, the server is able to operate on compressed map files, which
would then just take up the space of the compressed tarball they came in. \\
Since around SVN revisions r14859 to r14871 in July 2011 the compression related code has been removed from the source code; \\
saying that since 1.70.0 and beyond the server does not handle compressed maps files anymore.
The compression related code is found in //server/common/porting.c// :
/**
* This is a list of the suffix, uncompress and compress functions. Thus,
* if you have some other compress program you want to use, the only thing
* that needs to be done is to extended this.
* The first entry must be NULL - this is what is used for non
* compressed files.
*/
const char *uncomp[NROF_COMPRESS_METHODS][3] = {
{ NULL, NULL, NULL },
{ ".Z", UNCOMPRESS, COMPRESS },
{ ".gz", GUNZIP, GZIP },
{ ".bz2", BUNZIP, BZIP }
};
And the functions are :
* ''static FILE *open_and_uncompress_file(const char *ext, const char *uncompressor, const char *name, int flag, int *compressed, const char *mode)''
* ''FILE *open_and_uncompress(const char *name, int flag, int *compressed, const char *mode)''
* ''void close_and_delete(FILE *fp, int compressed)''
To find out, if the server actually is able to open compressed map files, \\
would be to compress the early map file HallOfSelection \\
and probably the files in the //start// folder. \\
On Linux it would be
* ''compress HallOfSelection'' to create a .Z file
* ''bzip2 HallOfSelection'' to create a .bz2 file
* ''gzip HallOfSelection'' to create a .gz file
Either the server keeps going, or would abort and send a message to the logfile as
>[Error] Can't open SRC_PATH/crossfire-1.70.0/_install/share/crossfire/maps/HallOfSelection: No such file or directory
>[Error] Initial map /HallOfSelection can't be found! Please ensure maps are correctly installed.
>[Error] Unable to continue without initial map.
and when creating a link //HallOfSelection// to point to the //HallOfSelection**.gz**// writing to the logfile
>[Error] Error loading map header - did not find a newline - perhaps file is truncated? Buf=BZh91AY&SY[GARBLED_STREAM]
>[Error] Error loading map header for /HallOfSelection, flags=0
>[Error] Initial map /HallOfSelection can't be found! Please ensure maps are correctly installed.
>[Error] Unable to continue without initial map.
>[Error] Can't open SRC_PATH/crossfire-1.70.0/_install/share/crossfire/maps/HallOfSelection: No such file or directory
>[Error] Initial map /HallOfSelection can't be found! Please ensure maps are correctly installed.
>[Error] Unable to continue without initial map.
----
===== Compression ==
When the server supports compressed map files, and it is desired to use compressed map files, \\
then some code has to filter out files inside the maps directory, \\
that are better left uncompressed. \\
:!: By all means, any .sh, .pl and .py script files need to be untouched.
The below bash code assumes that the script needs to run from a controlling terminal, \\
and that it would be located in the top level of the maps directory among the \\
many folders and files like HallOfSelection , regions | regions.reg , HallOfDMs . \\
Furthermore it omits files, that are (permanent) apartment, private shop and guilds related. \\
Of course, both //gzip// commands can be replaced by the desired compressor; f.ex //compress// or //bzip2// .
In the below code **gzip** had been selected, since it allows easy recursive decompression of directories by ''gunzip -r *'' .
Jump to [[#End Compression]] .
A shell code script to compress the map file using //gzip// could look as
#!/bin/bash
which gzip || exit
tty || exit # exit if clicked on it in the filer
find . -type f \( -not -name README \
-a -not -wholename "*.svn/*" \
-a -not -wholename "*python/*" \
-a -not -wholename "*templates/*" \
-a -not -wholename "*editor/*" \
-a -not -wholename "*test/*" \
-a -not -wholename "*styles/*" \
-a -not -wholename "*Info/*" \
-a -not -wholename "*/*apartment*" \
-a -not -wholename "*/*Apartment*" \
-a -not -wholename "*/*APARTMENT*" \
-a -not -wholename "*/*apart*" \
-a -not -wholename "*/villa/*" \
-a -not -wholename "*/guild/*" \
-a -not -wholename "*/guilds/*" \
-a -not -whonename "*/pshops/*" \
-a -not -wholename "*/pshop*" \
-a -not -wholename "*/privateshop*" \
-a -not -name "*.sh" \
-a -not -name "*.pl" \
-a -not -name "*.py" \
-a -not -name "*.gz" \
-a -not -name "*.bz2" \
-a -not -name "*.Z" \
-a -not -name "luxhouse" \
-a -not -name "housebrxzl" \
-a -not -name "keysale" \
-a -not -name "cdcapart*" \
-a -not -name "guild*" \
-a -not -name "*_lounge" \
-a -not -name "storage_hall*" \
-a -not -name "bigchest" \
-a -not -name "basement" \
-a -not -name "mainfloor" \
-a -not -name "secondfloor" \
-a -not -name "hallofjoining" \
-a -not -name "privateshop*" \
-a -not -name "ChangeLog" \
-a -not -name "Copying" \
-a -not -name "regions*" \) \
-exec gzip {} \;
== End Compression ==
Jump up to [[#Compression]] .
----
===== Bugs and Patches ==
The code has a flaw even until version 1.60.0 of the server trying to rename(filename, final), when their names \\
are identical. A bit further down, both file names get a different name by TEMP_EXT ".savefile" , so we need to \\
put that code there also in file //server/common/map.c// function //save_map()// :
if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
char buf[MAX_BUF];
snprintf(buf, sizeof(buf), "%s > %s%s", uncomp[m->compressed][2], filename, TEMP_EXT);
snprintf(final, sizeof(final), filename);
+ snprintf(filename, sizeof(filename), "%s%s", final, TEMP_EXT);
fp = popen(buf, "w");
And probably use strcmp 2 times near and at the very end:
--- 1411,1461 ----
} else {
fflush(fp2);
fclose (fp2);
unlink(final_unique); /* failure isn't too bad, maybe the file doesn't exist. */
!
! if ( ( strcmp(buf, final_unique) != 0 ) && rename(buf, final_unique) == -1) {
! LOG(llevError, "new_save_map:Couldn't rename unique file %s to %s\n", buf, final_unique);
! if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
! pclose(fp);
! } else {
! fclose(fp);
! }
return SAVE_ERROR_URENAME;
}
chmod (final_unique, SAVE_MODE);
}
unlink(final); /* failure isn't too bad, maybe the file doesn't exist. */
! if (rename(filename, final) == -1) {
! LOG(llevError, "Couldn't rename regular file %s to %s\n", filename, final);
return SAVE_ERROR_RRENAME;
unlink(final); /* failure isn't too bad, maybe the file doesn't exist. */
! if ( (strcmp(filename, final) != 0) && rename(filename, final) == -1) {
! LOG(llevError, "save_map:Couldn't rename regular file %s to %s\n", filename, final);
return SAVE_ERROR_RRENAME;
==== 1.11.0 ==
The code is faulty in version 1.11.0 of the server, not pclose'ing opened .savefiles. \\
The main errors are fixed in at least version 1.50.0, so it is a little bit disturbing, \\
why the compression code had been removed since 1.70.0 .
Basically it is about backporting the code from v.1.50.0 .
---
4 times pclose or fclose in **new_save_map()** ( common/map.c ) before early return from the function; \\
otherwise hundreds of shell commands compress will still be sleeping running and visible in a terminal through **ps** output:
--- 1359,1402 ----
if ((flag == SAVE_MODE_NORMAL || flag == SAVE_MODE_OVERLAY) && !m->unique && !m->template) {
char final_unique[MAX_BUF];
snprintf(final_unique, sizeof(final_unique), "%s.v00", create_items_path (m->path));
snprintf(buf, sizeof(buf), "%s%s", final_unique, TEMP_EXT);
+ LOG(llevDebug, "new_save_map:Saving unique map from %s to %s ...\n", buf, final_unique);
+
if ((fp2 = fopen (buf, "w")) == NULL) {
! LOG(llevError, "new_save_map:Can't open unique items file %s\n", buf);
! if (m->compressed && (m->unique || m->template || flag != SAVE_MODE_NORMAL)) {
! pclose(fp);
! } else {
! fclose(fp);
! }
return SAVE_ERROR_UCREATION;
**Note**: m->template became m->is_template in version 1.50.0
---
---
**res** needs to be set beforehand in **save_objects()** ( common/map.c ) , otherwise would return wronly :
int save_objects (mapstruct *m, FILE *fp, FILE *fp2, int flag) {
! int i, j = 0,unique=0, res=0;
object *op, *otmp;
+
/* first pass - save one-part objects */
for(i = 0; i < MAP_WIDTH(m); i++)
for (j = 0; j < MAP_HEIGHT(m); j++) {