Saving save games

•15. September 2014 • Leave a Comment

This text has been written by the admin of un.nethack.nu, regarding the recent move of the public servers onto new machines. You can also read it on his blog.

Long post so TL;DR for lazy people:

Patched 47 binaries to achieve backwards compatibility with old saves

I decided to redo the setup for un.nethack.nu mostly from scratch, to make it easier to support shared state between multiple servers. For historical reasons unnethack ran as uid 5. This uid belongs to the “games” user in debian, which the original server ran. Nowadays I prefer CentOS where uid 5 by default belongs to the username “sync”.

So I took the opportunity to use a dedicated user with a non-system uid. This became a pretty big issue since nethack also writes the uid to the save file and if it doesn’t match what it’s running as when you restore it’ll helpfully tell you the save file is not yours and delete it.

Brainstorming in the unnethack IRC channel a few options came up:

  1. Move back to uid 5
  2. Recompile all the binaries without the check
  3. Remove the uid check in the binaries
  4. Try to replace the uid inside the save files

The first is the ugly solution, which I see as a last resort. Investigation of the code writing the save files found that the uid was being written to a non-fixed position. This makes it pretty risky to mess with it. Removing the uid check and recompiling was done for the latest binary. Unfortunately there are currently 48 different versions of unnethack with version dependent saves. Even if I had some kind of tag or reference to the point in history where each binary was built I wouldn’t rely on being able to get the exact same binary. So investigation began on the fourth option.

Finding the code that does the uid check was easy, just need to look for the string being printed when it happens, “Saved game was not yours.”. The uid check and printing seems to be done in “restgamestate” in restore.c:

  STATIC_OVL
  boolean
  restgamestate(fd, stuckid, steedid)
  register int fd;
  unsigned int *stuckid, *steedid;  /* STEED */
  {
      /* discover is actually flags.explore */
      boolean remember_discover = discover;
      struct obj *otmp;
      int uid;
  
      mread(fd, (genericptr_t) &uid, sizeof uid);
      if (uid != getuid()) {    /* strange ... */
        /* for wizard mode, issue a reminder; for others, treat it
           as an attempt to cheat and refuse to restore this file */
        pline("Saved game was not yours.");
  #ifdef WIZARD
        if (!wizard)
  #endif
          return FALSE;
      }
  // More code...

And restgamestate is being called by dorecover (strange name for a function that restores save files…), also in restore.c:

  int
  dorecover(fd)
  register int fd;
  {
      unsigned int stuckid = 0, steedid = 0;  /* not a register */
      xchar ltmp;
      int rtmp;
      struct obj *otmp;
  
  #ifdef STORE_PLNAME_IN_FILE
      mread(fd, (genericptr_t) plname, PL_NSIZ);
  #endif
  
      restoring = TRUE;
      getlev(fd, 0, (xchar)0, FALSE);
      if (!restgamestate(fd, &stuckid, &steedid)) {
          display_nhwindow(WIN_MESSAGE, TRUE);
          savelev(-1, 0, FREE_SAVE);    /* discard current level */
          (void) close(fd);
          (void) delete_savefile();
          restoring = FALSE;
          return(0);
      }
      
  // More code...

So if the uid check in “restgamestate” fails it will return FALSE and cause “dorecover” to delete the save file. It seems the easiest solution would be to remove the “return FALSE” in the uid check. Now we need to find this part in the binary. The best and easiest way I found to disassemble in linux is to use objdump:

    $ objdump -D -S -g -t -T unnethack.45 | less -i

I started by searching for calls to getuid and looking at what function they were in. After a number of hits I find a call to it in “dorecover”:

000000000050cf10 <dorecover>:
  .....
  50cf4f:  e8 fc ef ff ff   callq  50bf50 <mread>
  50cf54:  44 8b 64 24 14   mov    0x14(%rsp),%r12d
  50cf59:  e8 82 14 0f 00   callq  5fe3e0 <__getuid>
  50cf5e:  41 39 c4         cmp    %eax,%r12d
  50cf61:  74 19            je     50cf7c <dorecover+0x6c>
  50cf63:  31 c0            xor    %eax,%eax
  50cf65:  bf 06 48 6a 00   mov    $0x6a4806,%edi
  50cf6a:  e8 21 f3 fd ff   callq  4ec290 <pline>
  .....

But getuid isn’t called in dorecover. The nearby calls (mread, pline) actually looks like the code from restgamestate. Lets see what’s being put in the edi register by the mov instruction before the call to pline by looking at the string data in the binary:

    $ objdump -s -j .rodata unnethack.45 | less -i

Searching for ” 6a48″ finds the place (6a4800 and then 6 characters in):

  6a4800 74202564 21005361 76656420 67616d65  t %d!.Saved game
  6a4810 20776173 206e6f74 20796f75 72732e00   was not yours..

So it is the code we’re looking for, it must have been inlined. Let’s analyze what that part actually does:

000000000050cf10 <dorecover>:
  .....
  50cf4f:  e8 fc ef ff ff       callq  50bf50 <mread>           # read piece of the file
  50cf54:  44 8b 64 24 14       mov    0x14(%rsp),%r12d         # move into %r12d (i think)
  50cf59:  e8 82 14 0f 00       callq  5fe3e0 <__getuid>        # call getuid
  50cf5e:  41 39 c4             cmp    %eax,%r12d               # compare result to %r12d
  50cf61:  74 19                je     50cf7c <dorecover+0x6c>  # if the same, jump to 50cf7c -|
  50cf63:  31 c0                xor    %eax,%eax                # set %eax to 0                |
  50cf65:  bf 06 48 6a 00       mov    $0x6a4806,%edi           # 6a4806: "Saved game was not yours."
  50cf6a:  e8 21 f3 fd ff       callq  4ec290 <pline>           # print it                     |
  50cf6f:  80 3d 94 4c 42 00 00 cmpb   $0x0,0x424c94(%rip)      # 931c0a <flags+0xa> check if field at flags + 0xA is equal to 0 (most likely WIZARD flag)
  50cf76:  0f 84 66 03 00 00    je     50d2e2 <dorecover+0x3d2> # if so, jump to 50d2e2 -|     |
  50cf7c:  ba d0 00 00 00       mov    $0xd0,%edx               #                        |   <-| Here!
  50cf81:  be 00 1c 93 00       mov    $0x931c00,%esi           #                        |
  50cf86:  89 df                mov    %ebx,%edi                #                        |
  50cf88:  e8 c3 ef ff ff       callq  50bf50 <mread>           #                        |
                                                                #                        |
  .....                                                         #                        |
                                                                #                        |  
  50d2e2:  be 01 00 00 00       mov    $0x1,%esi                #                      <-| Here! 
  50d2e7:  8b 3d 47 dd 41 00    mov    0x41dd47(%rip),%edi      # 92b034 <WIN_MESSAGE>
  50d2ed:  ff 15 ed 0e 45 00    callq  *0x450eed(%rip)          # 95e1e0 <windowprocs+0x60>
  50d2f3:  ba 04 00 00 00       mov    $0x4,%edx
  50d2f8:  31 f6                xor    %esi,%esi
  50d2fa:  bf ff ff ff ff       mov    $0xffffffff,%edi
  50d2ff:  e8 3c 40 00 00       callq  511340 <savelev>
  50d304:  89 df                mov    %ebx,%edi
  50d306:  e8 35 3b 10 00       callq  610e40 <__libc_close>
  50d30b:  e8 e0 cd f5 ff       callq  46a0f0 <delete_savefile>
  50d310:  c6 05 69 01 43 00 00 movb   $0x0,0x430169(%rip)      # 93d480 <restoring>
  50d317:  48 83 c4 20          add    $0x20,%rsp
  50d31b:  31 c0                xor    %eax,%eax
  50d31d:  5b                   pop    %rbx
  50d31e:  5d                   pop    %rbp
  50d31f:  41 5c                pop    %r12
  50d321:  c3                   retq

The patch needed turned out to be very simple, change the jump at 50cf61 to an unconditional one.

Looking at an x86 opcode and instructions reference we can find that the JMP (unconditional jump) opcode of the same type as the JE (opcode 0x74, rel8, jump relative to position) is 0xEB. Looking through a few unnethack binaries the disassembled output is very similar, following a pattern for 32-bit and another for 64-bit. So I wrote a fuzzy patcher:

    #!/usr/bin/env python
    import sys
    import subprocess
    
    patch_64 = {"needle": [0x44, '*', '*', '*', '*',
                           0xe8, '*', '*', '*', '*',
                           0x41, '*', '*',
                           0x74, 0x19,
                           0x31, 0xc0,
                           0xbf],
                "pos": 13,
                "new": [0xEB]}
    patch_32 = {"needle": [0x8b, '*', '*',
                           0xe8, '*', '*', '*', '*',
                           0x39, 0xc3,
                           0x74, 0x19,
                           0xc7],
                "pos": 10,
                "new": [0xEB]}
    
    def run(cmd):
        return subprocess.Popen(cmd,
                                stdout=subprocess.PIPE).communicate()[0]
        
    def main():
        if len(sys.argv) == 1:
            print("usage: patcher.py <file>")
            return
    
        fn = sys.argv[1]
        print("opening %s" % fn)
    
        output = run(['file', fn])
        if output.find('64-bit') > -1:
            print("64-bit binary")
            patch = patch_64
        elif output.find('32-bit') > -1:
            print("32-bit binary")
            patch = patch_32
        else:
            raise Exception("Unknown binary: %s" % output)
        
            
        f = open(fn, 'r+b')
        b = f.read(1)
        fpos = 1
        npos = 0
        needle = patch["needle"]
        spos = -1
        while not b == "":
            b = ord(b)
            if npos == len(needle):
                print "found it"
                if not spos == -1:
                    raise Exception("Expected to find pattern only once")
                spos = fpos - 1
                npos = 0
            if needle[npos] == '*' or needle[npos] == b:
                npos = npos + 1
            else:
                npos = 0
            b = f.read(1)
            fpos = fpos + 1
    
        if spos == -1:
            raise Exception("Could not find pattern")
    
        f.seek(spos - len(needle) + patch["pos"])
        b = ord(f.read(1))
        assert b == needle[patch["pos"]]
    
        print("writing patch")
        f.seek(spos - len(needle) + patch["pos"])
        for b in patch["new"]:
            f.write(chr(b))
    
        print("done!")
        
    if __name__ == "__main__":
        main()

This worked for all but one binary, where the compiler had not inlined the restgamestate call. It was easily patched manually, with the help of emacs wonderful hexl-mode to hexedit the binary.

New US Public Server (us.un.nethack.nu)

•15. September 2014 • 1 Comment

We have once again a US based public server at us.un.nethack.nu, thanks to Christian Stegen who is paying for the machine and our admin octe who has setup the server.

The european based server is now reachable at eu.un.nethack.nu and the ttyrec and dumps of both servers are consolidated on the website un.nethack.nu.

Both servers are now also reachable by passwordless SSH, using the user “unnethack”.

Junethack 2014 – Post Mortem

•27. August 2014 • Leave a Comment

This is the post-mortem for Junethack 2014.

The Junethack site was broken on the opening day because of a stupid last minute hotfix. The developer responsible for this has been punished with writing a PHP project not under 10000 lines of code.

The addition of dNetHack one week after the start was easy to do, also because of the developer being very cooperative and forthcoming.

In the UnNetHack and NetHack4 forks several bugs were found and fixed. AceHack and GruntHack were used this year for extensive unique death message generation.

Clan overcaffeinated and demilichens fought hard over the “unique death” clan trophy. In the end, it was a close call for demilichens to win in that category and also overall, as clan Justice was really close on their heels for the clan competition win.

We don’t offer any means of communicating between users on the site itself and this is intentional. We want them to self-organize and this year this went as far that clan overcaffeinatede and demilichens reached a mutual agreement not exploit some easily achievable but highly boring to do deaths for the “unique death” clan trophy.

Members of both clans wrote up their impressions on the tournament and gave some suggestions on how to improve the unique death trophy. This will certainly have an impact on how the unique death normalization will look like next year.
http://74.135.83.0:8018/nethack-stuff/junethack-unique-deaths-comments-2014-jonadab.html
http://mathematicalcoffee.blogspot.com.au/2014/07/junethack-2014-is-over.html

Another piece of feedback by a player can be read here:
http://www.reddit.com/r/nethack/comments/29m6wz/next_year_take_part_in_junethack_it_has_been/
And now for the boring statistics part:

195 players registered on the server, 146 linked their account with the public servers, and 119 actually played at least one game (last year: 157, 134, and 118).

6416 games were played on all 9 public servers during the tournament by registered users, 12323 were played by all players including those not taking part in the tournament (last year: 5210, 10, and 12313).

14627 games (including games by not registered players) fell into our start scum filter. This is lower than last year (20855) and much lower than the all-time high of 171760 from 2012.

50 different players ascended a total of 154 games (last year: 60 and 173).

Tournament games by variant (numbers from 2013 in brackets):

NetHack 3.4.3: 2111 (2849)
SporkHack: 164 (134)
UnNetHack: 811 (975)
AceHack: 1298 (520)
GruntHack: 1411 (574)
NetHack4: 111 (158)
DNetHack: 221 (-)
NetHack 1.3d: 289 (-)

Detailed info can be found on these three pages:
https://junethack.de/activity
https://junethack.de/ascensions
https://junethack.de/scoreboard

There are some minor improvements coming up for the next tournament. This year has shown again that the clan management code is in serious need of a refactoring, the user home page has too much information available elsewhere and some of the other pages have become so long that they are hard to navigate. Hopefully we’ll have good solutions for these problems by next year.

Thanks for playing, see you next year!

Junethack 2014 – the 4rd NetHack Cross-Variant Summer Tournament

•1. June 2014 • Leave a Comment

The fourth installment of the annual NetHack Cross-Variant Summer Tournament called Junethack has started on Sunday June 1st 2014 at midnight UTC and will be running until June 30 midnight UTC.

This tournament is trying to appeal not only to the hardcore serial ascenders but also to players that get constantly mangled and beaten to death in unrealistic brutal situations by this sadistic game (that means probably you) by offering various non-winning achievements and encouraging people to try and play NetHack forks.

This year, we have again NetHack 1.3d as part of the tournament. The UnNetHack and NetHack4 forks had updates since last year, so be on the look-out for dangerous changes in those games ;-).

You participate in the tournament by playing Vanilla NetHack, SporkHack, UnNethack, AceHack, GruntHack, NetHack4, or NetHack 1.3d on the supported public servers and linking those accounts on your Junethack user page.

You can get in contact with the organizers of Junethack here, on reddit, on rec.games.roguelike.nethack, and in the #junethack IRC channel on the Freenode IRC network.

If you want to help out in the development of the tournament, you can check out our code from GitHub.

Register at the tournament homepage and start venturing into the Dungeons of Doom!

Update 07 Jun 2014

DNetHack has been added as a supported variant to the tournament. You can play it at dnethack.ilbelkyr.de. Take note that now the cross variant achievements take one more variant to get. Although those players that already got those achievements will not lose them.

UnNetHack 5 Android port on the Google Play Store

•19. January 2014 • 2 Comments

UnNethack 5 is now available on the Google Play Store.

This version currently features an ASCII view of the map, the standard unchozo32b tileset and the brand new DawnHack tileset. Future versions will include the Geoduck and Absurd tileset.

Android UnNetHack with ASCII interface

Showing the map in ASCII

UnNetHack 5 – “Doing Releases like a pro” released!

•2. January 2014 • 16 Comments

Announcing the release of UnNetHack 5.

Source code, Debian package and Windows TTY and GUI executables can be downloaded here:
http://sourceforge.net/projects/unnethack/files/unnethack/5.1.0/

This is a major release, more than one and a half years in the making and therefore has an awful lot of changes and new features. So here is only a summary, the (mostly) complete ChangeLog is in the download packages or can be read on-line at https://sourceforge.net/p/unnethack/code/HEAD/tree/tags/5.1.0-20131208/ChangeLog

The last release was originally planned to be released after the end of this year’s Junethack but this deadline was missed and then the amount of commits skyrocketed. This was also due to the increase of UnNetHack’s DevTeam. We are now more than ever and are not going away and stop doing releases.

Two new branches have been added: The winter-themed “Sheol”, filled with tons of new, strange monster and dangers and the orc-infested “Ruins of Moria”.

Some first attempts at increased role and race differentiation has been incorporated in this version:

  • Tourists get automatic type identification for shop items
  • Healers can see how wounded monsters are
  • Knights get a weight bonus for body armor heavier than studded leather armor
  • Archeologists can enchant fedoras to +7
  • Elven/vampiric players do not regenerate health while touching iron/silver with bare skin (respectively)

Several long standing annoyances have been approached:

  • Sokoban luck penalty has been removed and instead tracking of solving Sokoban without any tricks has been added
  • The Quest turn limit has been removed
  • The probability of the Fort Ludios portal was increased; it exists now in around 93% of the games
  • Only the Sanctum and the Astral Plane are unmappable levels
  • New paranoid options have been added, asking for confirmation when walking into known lava and water squares
  • The Valley of the Dead features several portals leading to Vlad, the Dragon Caves, and Sheol, making navigating Gehennom less tedious
  • Dragons auto-ID after observing breath attacks or after being probed

New conducts:

  • Permanent hallucination conduct option: perma_hallu
  • Disable death drops conduct option: deathdropless
  • Disable Elbereth conduct option: elberethignore
  • A new menu for choosing conducts at the beginning of the game (currently only TTY)

And of course lots of bug fixes.

Happy Hacking!

Reddit Roguelike Challenge 6: Nethack, UnNethack and SporkHack

•2. December 2013 • Leave a Comment

The current Reddit Roguelike Challenge on Reddit is about NetHack forks.

Post there how far you got and what score you achieved or tell the story of your gruesome death.

Happy hacking!

 
Follow

Get every new post delivered to your Inbox.

Join 157 other followers