User:Pi Masta
Alright I finally created this page. Tired of seeing so much red on the Recent Changes page.
Um, I like computers and math (hence Pi) and of course XCOM or I wouldn't be here. Just something about the gameplay and concept of the game intrigues me.
PyXcom
I'm hoping to get a possible 'free-for-all' editor going in python. I had success getting an old version of Python to work in DOSBox, though it's not very stable. I'm working on a 'back-end' (currently dubbed PyXcom) for all the saved game files so applications would just have to worry about what values to put in and not formatting the file correctly, giving it a more 'pythonic' interface. Could still use work, but I have a simple class that can read and write .DAT files (really any fixed-length files), and is somewhat simple to add new fields as values in files are discovered.
From this I could make a Tk/Tcl version that would require a much later version of Python to do editing on modern machines (including *nix, and probably Mac). This would probably do more in depth editing, of say the bases, stats of objects or soldiers, etc.
Before that I think I'll make a DOS friendly version (aka command prompt, maybe keyboard selection) that could implement changes (I'm thinking of making a better soldier equip screen that runs before going into tactical, downside is I don't it will be easy to display images).
Another idea I have is to make a Monitoring program (PyXMon?) that can watch the save game and MISSDAT directories and apply patches to them automattically. The main use this would have is renaming soldiers since it is impossible to have an App, like XcomUtil,rename them after combat (GEOSCAPE updates the stats then), instead you could save your game, possibly wait a second or 2 and then reload it. The monitoring program will catch the new save game and apply the fix. This would also go with the 'Finished' Dirt Modules bug, though handling that between battles would probably be enough. Another application would be for Data Miners to have the scripts automattically output data from files (in a more readable format) on saves, for instance Zombie's initial Funds data could have had a script written in python to automatically append a file with the data about the countries.
I have too many ideas, and most likely I won't complete these projects, but I hope to get close. I'm updating this Wiki for the most part on the files, and trying to crack some of the others.
Current Status
I have almost all decoded files supported (with the exception of graphic files which I don't intend to directly load) and I've started on making a layer on top of it. Namely BaseInfo, SoldierInfo, LocInfo, and CraftInfo that can all be connected together and I have made a 'master' module to do this given a directory for it to load from. Let me give some examples:
>>> t = Geoscape('missdat')
>>> t.base[0].name
'Main'
>>> for x in t.base[0].soldiers:
print x
D Likhachev (r)aBRst (Rookie)
F Robinson (r)aRbr (Rookie)
Evans TU (C)C-AC*ST* (Captain)
.......
Sarah Watson (r)arbrst (Rookie)
Ed Kemp SACK (Rookie)
>>> for x in t.loc.used:
print x, ':', x.data
Loc[0] - XCOM Base : Base 0: Main
Loc[1] - XCOM Ship : Skyranger-1
Loc[2] - XCOM Ship : Interceptor-1
Loc[3] - XCOM Ship : Interceptor-4
Loc[4] - XCOM Base : Base 1: Asia
Loc[5] - Crash Site : Large Scout-51
Loc[6] - XCOM Base : Base 2: America
Loc[7] - XCOM Ship : Interceptor-5
...
Loc[15] - Alien Base : None
>>> t.loc[5].data.damage #The crashed large scout 150
>>> for x in t.loc[1].data.soldiers: #Skyranger 1's soldiers
print x, ';',
D Likhachev (r)aBRst (Rookie) ; F Robinson (r)aRbr (Rookie) ; Evans TU (C)C-AC*ST* (Captain) ; Robinson TU (Co)ACrbr*ST* (Colonel);
C Dodge (S)*AC*Rbrst (Seargent) ; Yuzo Kojima SACK (Rookie) ; K Okamoto (r)W-ACr (Rookie) ; Yataka Shoji (r)C-st (Rookie) ;
Austin King SACK (Rookie) ; Y Matsumara (r)C-W-ACR (Rookie) ;
And now to show that you can chain these together. This one get's the second LOC.DAT entry (which is skyranger-1), grabs its data (the skyranger craft itself), gets the craft's second soldier (remember is all 0-based), then his/her base, that base's location, that location's data (which just goes back to the base), that base's last soldier, and then that soldier's name.
>>> t.loc[1].data.soldiers[1].base.loc.data.crafts[0].soldiers[-1].name 'Y Matsumara (r)C-W-ACR'
This is all just the tip of the iceberg!
Right now I just have mostly Geoscape stuff made for use. The Tactical saves can be manipulated but there's no referencing other files or what not. So far there is no GUI. This is intentional as it is mean to run at the command prompt with XCOM if possible. I might whip up a generated GUI using tkinter, one that just makes fields based on the information from these class objects (read: not going to look pretty). But right now I think I'm going to get on to the Tactical files (the main ones, OBREF, UNITREF, etc).
Oh and I do have a stat string generator that is very customizeable, although probably not as simplistic.
I also now have a simple graphical soldier equipment utility to possibly replace the ingame initial equipment screen.
As a tool
I've whipped up a handy file monitoring script that will run a function when a file is modified. With this I can have it automatically apply patches or fix issues whenever a game is saved. Also it could be possible with modification of the batch file that switches between Tactical and Geoscape to have it add or even replace parts of the game. (I'm trying to get a soldier editor, but there's a few issues I have to work out)
Mainly though, I can use this monitoring process to help figure out what certain bytes mean. For example when I was looking for what offset 12 does for OBPOS.DAT, I simply monitored one of my save folder's obpos.dat file. Whenever it changed my script would parse the information and compare it to the last known information (rather quickly too might I add), it then printed to a seperate console the results as well as appending it to a log file. I even added more functionality by having it read the SAVEINFO.DAT file to get the save name, thus I could put some comments on what I did while I saved. Put this all together and I could see what happened to the files (or what parts I want to know about) as often as I saved, no need to alt-tab out and run a script.
LEASE.DAT
some working notes on lease.dat
fixed length of 180 bytes, prime factorization of 180:
2^2 * 3^2 * 5,
factors of 180: (probable line widths)
2, 3, 4, 5, 6, 9, 10, 12, 15, 18, 20, 30, 36, 45, 60, 90, 180
only offsets that seem to change (out of about 14 files) (compared from 1st to 14 others, but not 2nd or 3rd etc with the other 14 so could be more)
0-1, 5-17, 20-23, 26-28, 32-35, 38-41, 44-47, 50-53, 56-59, 62-65, 68-71, 74-77, 80-83, 86-89, 92-95, 98-101, 104-107, 110-113, 116-119, 122-125, 128-131, 134-137, 140-142, 146-149, 152-155, 158-161, 164-167, 170-173,
There is a lot of symmetry in this format, lots of double 0's will line up at the line widths of the factors, with the exception of the very first byte, almost as if the first record is somehow different...
- 13 Seems to be 0 for saves, 1 after combat and in missdat, 2 only for before combat, but I've seen this number change in Geoscape saves, particularly when intercepting craft
Speculation
LEASE as the name of this file makes me think of it pertaining to either:
- Purchased craft monthly (shouldn't need this much data)
- Base monthly costs
The 'orange' set of values seems to be the most popular and 'simple'.
From tests using PyXCOM of over 70 saves, this file seems to change most often when a UFO is detected or an interceptor is launched at it.
Base Items store space
I dug into GEOSCAPE.EXE and found the pattern of sizes like that listed on the stores page (offset 0x5DB34). However it different some areas, namely the HWP ammo was not with the actual HWPs. Also interesting was that there were gaps of 0's between some items, usually from the different groups like craft weapons, to agent weapons, to alien weapons, etc.
I got to looking at PURCHASE.DAT as this seemed to be the only logical place for size information to be. Of particular interest was the so called 'sort' order offset. I really doubted the programmers devoted a byte to help 'sort' the information for display, but given what could be gleaned from that file alone it was a good guess. I'm pretty sure it's an index (which technically does sort it) to at least one array (I surmise others below). I used some python scripts to read in a Purchase.dat file and a hex editor to find the pattern in geoscape.exe.
Simply by placing the values in an array and having each item in purchase.dat use it's 'sort' value to fetch the record in the array, everything matched up, well almost. The plasma beam and Fusion Ball Launcher were backwards (plasma had 20 space while fusion had 12). I almost gave up but then I thought what if the order is wrong? There isn't much info linking to what item a record is in that file, but I looked at the sell price. Turns out I was right, the sell price of the Fusion Ball Launcher is well known as it's the second runner up for Manufacturing Profitability.
Other Possibilities
These items have to index to English.dat somewhere (probably in the executable). My guess is that it is near the array that I found. There is probably other information such as the craft weapon statistics. The gaps between these indexes might make it possible to add new items to the game as well.
INTER.DAT
Took a stab at this file. I believe it deals with interceptions based on the name and that it changed whenever I attacked a UFO with an interceptor. However it is 568 bytes with a factorization of:
2 * 2 * 2 * 71
Which baffles me because 71, 142, or 248 seem like weird lengths. I don't know of anything relating to these numbers in X-COM.
I observed one set of changes that is interesting:
0246: 000 ( 0) -> 008 ( 8) 0248: 000 ( 0) -> 090 (5A) 0388: 000 ( 0) -> 008 ( 8) 0390: 000 ( 0) -> 090 (5A) 0530: 000 ( 0) -> 008 ( 8) 0532: 000 ( 0) -> 090 (5A)
This appeared right after I downed a UFO in Brazil (Snakeman Large Scout on Retaliation) along with some other changes, but this set caught my eye because of the repetition, but more when I found that 388 - 246 = 142, and 530 - 388 = 142, with the significance of 142 being one of the factors of file size (2*71 = 142 -> 142 * 4 = 568)
Also observed offsets 32 and 33 jump from 0's to 254 and 255 respectively and back. 30 and 31 changed somewhat but appears unrelated with the value of 32 and 33. So my guess is 4 sets of 142 bytes, with 71 2-byte values inside of each set.
Perhaps this file is suppose to save the state of interceptions as they happen so they could be restored upon loading (which we all know crashes the game). Perhaps an adjustment could be made to this file to allow it to work, just like with the Difficulty bug and
I might have to check this, but I think the maximum number of interceptions you can have minimized is 4, so possibly this file keeps track of each one. IGLOB.DAT.
Files to be Decoded
Here I'm going to list some files that need values to be decoded. Some of them it might not be possible as a lot of the values are set to 0. Namely the MISDATA, MISSION, MISSION2 as they all use the same format but are used at different stages of the game.
- MISDATA.DAT (and MISSION.DAT, MISSION2.DAT) - probably should combine into 1 page with sections detailing what is and is not in each name
- GEODATA.DAT - 92 Bytes, but we only have about 10 of them decoded. I think this is what is used to generate new battlescapes, but I think it's GEOSCAPE that actually generates the MAP.DAT file
- LOC.DAT - Some unknown values
- Little to nothing known
- XBASES.DAT - Alien Base Info? - Initial tests suggest not, maybe a constant file? Never changed upon a new Alien base being made
- INTER.DAT - Stores info on interceptions? - Pretty sure on this
- AKNOW.DAT - Perhaps what the Aliens Know, as in which bases for retals
- LEASE.DAT