User:Pi Masta

From UFOpaedia
Revision as of 00:52, 6 April 2007 by Pi Masta (talk | contribs) (INTER.DAT notes)
Jump to navigation Jump to search

Alright I finally created this page. Tired of seeing so much red on the Recent Changes page.


Um, I like computers and math (hence Pi) and of course XCOM or I wouldn't be here. Just something about the gameplay and concept of the game intrigues me.

PyXcom

I'm hoping to get a possible 'free-for-all' editor going in python. I had success getting an old version of Python to work in DOSBox, though it's not very stable. I'm working on a 'back-end' (currently dubbed PyXcom) for all the saved game files so applications would just have to worry about what values to put in and not formatting the file correctly, giving it a more 'pythonic' interface. Could still use work, but I have a simple class that can read and write .DAT files (really any fixed-length files), and is somewhat simple to add new fields as values in files are discovered.

From this I could make a Tk/Tcl version that would require a much later version of Python to do editing on modern machines (including *nix, and probably Mac). This would probably do more in depth editing, of say the bases, stats of objects or soldiers, etc.

Before that I think I'll make a DOS friendly version (aka command prompt, maybe keyboard selection) that could implement changes (I'm thinking of making a better soldier equip screen that runs before going into tactical, downside is I don't it will be easy to display images).

Another idea I have is to make a Monitoring program (PyXMon?) that can watch the save game and MISSDAT directories and apply patches to them automattically. The main use this would have is renaming soldiers since it is impossible to have an App, like XcomUtil,rename them after combat (GEOSCAPE updates the stats then), instead you could save your game, possibly wait a second or 2 and then reload it. The monitoring program will catch the new save game and apply the fix. This would also go with the 'Finished' Dirt Modules bug, though handling that between battles would probably be enough. Another application would be for Data Miners to have the scripts automattically output data from files (in a more readable format) on saves, for instance Zombie's initial Funds data could have had a script written in python to automatically append a file with the data about the countries.

I have too many ideas, and most likely I won't complete these projects, but I hope to get close. I'm updating this Wiki for the most part on the files, and trying to crack some of the others.

Current Status

I have a large number of files supported and I've started on making a layer on top of it. Namely BaseInfo, SoldierInfo, LocInfo, and CraftInfo that can all be connected together and I have made a 'master' module to do this given a directory for it to load from. Let me give some examples: 

>>> t = Geoscape('missdat')
>>> t.base[0].name
'Main'

>>> for x in t.base[0].soldiers:
        print x
D Likhachev (r)aBRst (Rookie)
F Robinson (r)aRbr (Rookie)
Evans TU (C)C-AC*ST* (Captain)
.......
Sarah Watson (r)arbrst (Rookie)
Ed Kemp SACK (Rookie)

>>> for x in t.loc.used:
        print x, ':',  x.data	
Loc[0] - XCOM Base : Base 0: Main
Loc[1] - XCOM Ship : Skyranger-1
Loc[2] - XCOM Ship : Interceptor-1
Loc[3] - XCOM Ship : Interceptor-4
Loc[4] - XCOM Base : Base 1: Asia
Loc[5] - Crash Site : Large Scout-51
Loc[6] - XCOM Base : Base 2: America
Loc[7] - XCOM Ship : Interceptor-5
Loc[8] - XCOM Ship : Skyranger-2
Loc[10] - XCOM Ship : Interceptor-6
Loc[11] - XCOM Ship : Interceptor-7
Loc[12] - XCOM Base : Base 4: Australia
Loc[13] - XCOM Base : Base 5: South America
Loc[15] - Alien Base : None

>>> t.loc[5].data.damage #The crashed large scout
150

>>> for x in t.loc[1].data.soldiers: #Skyranger 1's soldiers
        print x, ';',	
D Likhachev (r)aBRst (Rookie) ; F Robinson (r)aRbr (Rookie) ; Evans TU (C)C-AC*ST* (Captain) ; Robinson TU (Co)ACrbr*ST* (Colonel);
C Dodge (S)*AC*Rbrst (Seargent) ; Yuzo Kojima SACK (Rookie) ; K Okamoto (r)W-ACr (Rookie) ; Yataka Shoji (r)C-st (Rookie) ; 
Austin King SACK (Rookie) ; Y Matsumara (r)C-W-ACR (Rookie) ;

And now to show that you can chain these together. This one get's the second LOC.DAT entry (which is skyranger-1), grabs its data (the skyranger craft itself), gets the craft's second soldier (remember is all 0-based), then his/her base, that base's location, that location's data (which just goes back to the base), that base's last soldier, and then that soldier's name. 

>>> t.loc[1].data.soldiers[1].base.loc.data.crafts[0].soldiers[-1].name
'Y Matsumara (r)C-W-ACR'

This is all just the tip of the iceberg!

Right now I just have mostly Geoscape stuff made for use. The Tactical saves can be manipulated but there's no referencing other files or what not. So far there is no GUI. This is intentional as it is mean to run at the command prompt with XCOM if possible. I might whip up a generated GUI using tkinter, one that just makes fields based on the information from these class objects (read: not going to look pretty). But right now I think I'm going to get on to the Tactical files (the main ones, OBREF, UNITREF, etc).

Oh and I do have a stat string generator that is very customizeable, although probably not as simplistic.


As a tool

I've whipped up a handy file monitoring script that will run a function when a file is modified. With this I can have it automatically apply patches or fix issues whenever a game is saved. Also it could be possible with modification of the batch file that switches between Tactical and Geoscape to have it add or even replace parts of the game. (I'm trying to get a soldier editor, but there's a few issues I have to work out)

Mainly though, I can use this monitoring process to help figure out what certain bytes mean. For example when I was looking for what offset 12 does for OBPOS.DAT, I simply monitored one of my save folder's obpos.dat file. Whenever it changed my script would parse the information and compare it to the last known information (rather quickly too might I add), it then printed to a seperate console the results as well as appending it to a log file. I even added more functionality by having it read the SAVEINFO.DAT file to get the save name, thus I could put some comments on what I did while I saved. Put this all together and I could see what happened to the files (or what parts I want to know about) as often as I saved, no need to alt-tab out and run a script.

'GLOB' Files

I've noticed a lot of saved files end in 'GLOB', presumably short for Globe or Globals. 

 BGLOB.DAT - ambient lighting on tactical maps
 IGLOB.DAT - date and time
LIGLOB.DAT - Money (I also think it holds income,expenses, etc for graphs as well)
UIGLOB.DAT - Largely unknown, but holds scores for completed research?
 WGLOB.DAT - Tactical turns elapsed, number of aliens/soldiers/tanks/civvies kills as well as score

LEASE.DAT

some working notes on lease.dat

fixed length of 180 bytes, prime factorization of 180: 

2^2 * 3^2 * 5,

factors of 180: (probable line widths) 

2, 3, 4, 5, 6, 9, 10, 12, 15, 18, 20, 30, 36, 45, 60, 90, 180

only offsets that seem to change (out of about 14 files) (compared from 1st to 14 others, but not 2nd or 3rd etc with the other 14 so could be more) 

0-1,  5-17,  20-23,  26-28,  32-35,  38-41,  44-47,  50-53,  56-59,  62-65,  68-71,  74-77,  80-83,  86-89,  
92-95,  98-101,  104-107,  110-113,  116-119,  122-125,  128-131,  134-137,  140-142,  146-149,  152-155,  
158-161,  164-167,  170-173,

There is a lot of symmetry in this format, lots of double 0's will line up at the line widths of the factors, with the exception of the very first byte, almost as if the first record is somehow different...

On the fields that change, there doesn't appear to be many different values for them, usually only 2 or 3, with the exception of the first byte again.

Patterns from 14 files

Out of all 14 files I'm looking at there are only 11 variations which make offsets 0 and 6 special as they are the only fields to have 11 different observed values. There are only a couple of offsets that didn't fall into 2 categories perfectly, this is demonstrated by the background of the offsets. Orange values are only seen with other orange values, and yellow values the same. Thus if say offset 0 is 65 (orange) all the bytes after offset 14 will be the ones highlighted in orange. If there is no background then it has been seen with both orange and yellow.

NOTE There's a good chance I goofed something up reguarding the orange and yellow's on offsets 1, 7, 12, and 13

Fields marked with volatile I have noticed to change the most between consecutive saves, usually when a UFO is detected, but not always.

offset observed values # comments
0 65 37 234 108 143 177 181 22 215 61 53 11 Volatile
1 0 1 10 3 Volatile
5 77 141 2
6 32 197 71 234 235 145 51 24 56 186 138 11 Volatile
7 0 1 2 10 6 5 Volatile
8 13 200 2
9 196 4 2
10 94 0 2
11 248 0 2
12 120 208 90 180 14 5
13 0 1 2 3 Seems to be 0 for saves, 1 after combat and in missdat, 2 only for before combat, but I've seen this number change in Geoscape saves, particularly when intercepting craft
14 7 136 2
15 38 60 2
16 136 142 2
17 71 216 2
20 78 19 2
21 248 199 2
22 139 70 2
23 70 252 2
26 70 137 2
27 248 70 2
28 115 254 2
32 252 104 2
33 38 239 2
34 198 25 2
35 7 232 2
38 201 80 2
39 203 255 2
40 200 118 2
41 4 254 2
44 30 252 2
45 184 30 2
46 136 104 2
47 60 251 2
50 161 14 2
51 172 232 2
52 19 238 2
53 199 226 2
56 0 10 2
57 0 11 2
58 137 192 2
59 70 117 2
62 60 118 2
63 30 254 2
64 104 255 2
65 239 118 2
68 29 118 2
69 15 8 2
70 80 255 2
71 255 118 2
74 255 157 2
75 118 15 2
76 252 235 2
77 30 25 2
80 25 254 2
81 144 255 2
82 14 118 2
83 232 252 2
86 131 14 2
87 196 64 2
88 10 1 2
89 11 70 2
92 17 94 2
93 255 252 2
94 118 38 2
95 254 128 2
98 252 117 2
99 255 187 2
100 118 184 2
101 8 1 2
104 6 201 2
105 232 202 2
106 157 4 2
107 15 0 2
110 255 0 2
111 118 0 2
112 254 30 2
113 255 184 2
116 232 142 2
117 236 216 2
118 14 199 2
119 64 70 2
122 252 0 2
123 196 30 2
124 94 197 2
125 252 86 2
128 63 0 2
129 0 67 2
130 117 205 2
131 187 33 2
134 0 137 2
135 31 78 2
136 201 254 2
137 202 31 2
140 200 254 2
141 2 37 2
142 0 24 2
146 136 0 2
147 60 27 2
148 142 192 2
149 216 247 2
152 254 201 2
153 24 202 2
154 0 4 2
155 30 0 2
158 6 0 2
159 184 0 2
160 0 86 2
161 67 30 2
164 114 60 2
165 3 142 2
166 137 216 2
167 78 233 2
170 138 38 2
171 70 128 2
172 254 63 2
173 37 32 2
176 61 38 2
177 1 128 2
178 0 63 2
179 27 0 2

Offsets 1, 7, 12, 13

Comparing offsets 1, 7, 12, and 13 (the only offsets that will not 'determine' the values after offset 14 (and others)).

NOTE There's a good chance I goofed something up reguarding the orange and yellow's on offsets 1, 7, 12, and 13, most of it should be accurate though

1 7 12 13 Comments
0 0 14 1
0 0 180 0
0 1 90 0
0 10 90 0
0 10 120 0
0 2 120 0
1 2 180 0
10 6 14 1
10 6 208 2

Speculation

LEASE as the name of this file makes me think of it pertaining to either:

  • Purchased craft monthly (shouldn't need this much data)
  • Base monthly costs

The 'orange' set of values seems to be the most popular and 'simple'.

From tests using PyXCOM of over 70 saves, this file seems to change most often when a UFO is detected or an interceptor is launched at it.

The 2 'sets' of the files are probably a result from me having essentially 2 games mixed in with my saves (at different difficulty levels). I was wanting to try something else but I didn't want to loose my current status, and now it's rather hard to tell which is which.


Base Items store space

Currently the General Stores page lists the space needed for holding items. I couldn't find anywhere here where these values were found, so I assumed they were gathered empirically. However I knew they had to be stored somewhere together. Also I figured that they would be stored as simple integers and not some floating point number. I noticed that each 'size' would become an integer if you multiplied it by 10, so from that ammo clips of size 0.1 would be stored as 1.

I dug into GEOSCAPE.EXE and found the pattern of sizes like that listed on the stores page (offset 0x5DB34). However it different some areas, namely the HWP ammo was not with the actual HWPs. Also interesting was that there were gaps of 0's between some items, usually from the different groups like craft weapons, to agent weapons, to alien weapons, etc.

I got to looking at PURCHASE.DAT as this seemed to be the only logical place for size information to be. Of particular interest was the so called 'sort' order offset. I really doubted the programmers devoted a byte to help 'sort' the information for display, but given what could be gleaned from that file alone it was a good guess. I'm pretty sure it's an index (which technically does sort it) to at least one array (I surmise others below). I used some python scripts to read in a Purchase.dat file and a hex editor to find the pattern in geoscape.exe.

Simply by placing the values in an array and having each item in purchase.dat use it's 'sort' value to fetch the record in the array, everything matched up, well almost. The plasma beam and Fusion Ball Launcher were backwards (plasma had 20 space while fusion had 12). I almost gave up but then I thought what if the order is wrong? There isn't much info linking to what item a record is in that file, but I looked at the sell price. Turns out I was right, the sell price of the Fusion Ball Launcher is well known as it's the second runner up for Manufacturing Profitability.

Other Possibilities

These items have to index to English.dat somewhere (probably in the executable). My guess is that it is near the array that I found. There is probably other information such as the craft weapon statistics. The gaps between these indexes might make it possible to add new items to the game as well.


INTER.DAT

Took a stab at this file. I believe it deals with interceptions based on the name and that it changed whenever I attacked a UFO with an interceptor. However it is 568 to a factorization of: 

2 * 2 * 2 * 71

Which baffles me because 71, 142, or 248 seem like weird lengths. I don't know of anything relating to these numbers in X-COM.

I observed one set of changes that is interesting: 

0246: 000 ( 0) -> 008 ( 8)
0248: 000 ( 0) -> 090 (5A)
0388: 000 ( 0) -> 008 ( 8)
0390: 000 ( 0) -> 090 (5A)
0530: 000 ( 0) -> 008 ( 8)
0532: 000 ( 0) -> 090 (5A)

This appeared right after I downed a UFO in Brazil (Snakeman Large Scout on Retaliation) along with some other changes, but this set caught my eye because of the repetition, but more when I found that 388 - 246 = 142, and 530 - 388 = 142, with the significance of 142 being one of the factors of file size (2*71 = 142 -> 142 * 4 = 568)

Also observed offsets 32 and 33 jump from 0's to 254 and 255 respectively and back. 30 and 31 changed somewhat but appears unrelated with the value of 32 and 33. So my guess is 4 sets of 142 bytes, with 71 2-byte values inside of each set.

Perhaps this file is suppose to save the state of interceptions as they happen so they could be restored upon loading (which we all know crashes the game). Perhaps an adjustment could be made to this file to allow it to work, just like with the Difficulty bug and IGLOB.DAT.

Retrieved from "https://www.ufopaedia.org/index.php?title=User:Pi_Masta&oldid=11516"