Talk:Firing Accuracy Testing
Arg, I get an error when trying to open your bb_mapedit3.zip ... is it just me? (is there a new version of zip??)
Does your Editor only edit a given savegame? (Was that just the custom uniform mod you were talking about with Zombie that mods other things?) I don't want to muck with anything but a savegame without knowing about it (thrice burnt by XcomUtil!).
By the way, I am doing Firing Accuracy testing and seeing some interesting but odd numbers. Remember that ultra cool numeric tileset you made for explosion testing? I could really use something like that, but with a twist. Particularly I need some walls (in addition to ground) that reflect damage. Just a simple north wall is enough. The point is to see the spread of misses. I seem to recall there's only so much terrain storage space... I expect a lot more hits on the walls than ground; maybe a 9:1 ratio. It's also ok if the numbers "wrap around" because this will only happen in a heavily hit area, making it clear where it transitioned. What I'm after is a "silhoutte" of misses (including ones that hit ground). Start the counting with 1 and go as far as you can. If you want more details before starting, just ask.
If you don't have the time to do it, I understand totally. I'll still have plenty of results... there just won't be as much understanding of why they are the way they are.
- MikeTheRed 17:12, 7 November 2006 (PST)
Most likely your download got corrupted. Try it again. I haven't spotted any updates to the ZIP format in years, though you could try an update if all else fails.
Neither the uniform mod or the editer changes anything besides save files. Both however add the files MAN_A.SPK through to MAN_N.SPK, and modify save files so that they get used.
The tileset already contains walls for you to use. To implement them, either use my editer on one of your savegames which uses the tileset, or run Diashiva's Map Editer on the MAP files. In addition, you'll find the file "desert.exe" in your terrain folder, which allows you to easily modify the armor values of the tiles.
The walls aren't numbered, however, but they don't need to be if all you're testing is where the shots go. Set the armor to 1, and if the wall gets destroyed, you know it was a hit. Also keep in mind that the chances of the ground being hit depend on how high the shooter is in relation to the target.
- Bomb Bloke 23:34, 7 November 2006 (PST)
Your editor opens fine now, and I see my previous download was too small... apparently I didn't notice an incomplete download (blush). Thanks for the note re: what your apps change. Almost nothing except the savegame... good deal.
What I'm asking is specifically whether you might be able to make numbered walls. Like you did for ground. So that I can see not just whether they're hit, but how often. (I meant a distribution spread, sorry.) Having both numbered walls and ground would be best. Walls only would be best, if only one can be done. But I repeat, it's not critical. Basically it will probably help in making a precise formula governing in-game accuracy... but it won't change the actual results, which are more than enough to see how it works in general. (Actual accuracy can be very different from the stated accuracy, in complex ways.)
- MikeTheRed 15:13, 8 November 2006 (PST)
I'm sorry, but this is one thing the game engine won't let me pull off. The reason is that when a tile is hit with a projectile, it takes a random amount of damage - 25% to 75% of the weapons capabilities.
The "armor" of each wall would need to be at most equal to that 25% mark, to ensure that each and every shot made a mark. However, if the dice roll comes out at 75%, then the tile will switch three times, not just the once - Making it impossible to get any trustworthy results after the very first shot had been made!
In case that that doesn't make any sense, I'll simplify it somewhat: At best, tile counters will go up by a random amount, from 1 to 3, with each shot. Therefore, to get any accuracy, you'd need to make a shot, then reload the map, and repeat - Making any shot counter system that counts above one, redundant.
I've crafted a suitable map, and the logger will be the work of just a few minutes. However, I'm stumped as to which format to use to save the results. Usually, I go for comma seperated data, but in this case the data is in the form of co-oridinates...
If there's some sort of 3D vector program that would be suitable to display the lines of fire, then it'd be great if I could get hold of the file format. Otherwise, I'll just dump the co-ords any old how, and write a program to show minimum/maximum/average/etc firing angles after.
- Bomb Bloke 20:42, 10 November 2006 (PST)
Hmm, well, I had been thinking of using weapon strength 1 and wall armor strength 1... the wall would take a hit half of the time, and I'd just take into account that I was only seeing half the hits, on average. It's much better than nothing, even if it's not a perfect counter.
What I want to see is things like, how wide the shots miss by, whether misses are equally spread from farthest left to farthest right (like, a simple random roll for the angle for misses), or whether they're concentrated near the intended target, or what. There appear to be two components to accuracy, shots that are rolled as hits, and shots that are rolled as misses; the question is the degree to which a miss roll actually hits because it's random-miss angle happens to intercept the target anyway. So the equation for predicting hit percent has at least two terms, and may need more... there may also be percent of rolled hits that actually miss, in addition to this apparent percent of rolled misses that actually hit. All of which varies by distance, of course.
For this, then, I would've liked a "wall counter" map, even if it's not perfect. By the way, I really doubt I'd need to count to more than 100 for any given tile or wall, given my setup of ammo count etc. If you can do the wall, I'd still be very interested in it.
I'm intrigued by you other suggestion, but am not sure exactly what your logger will be counting. Anything that would require me to save and reload after each shot would greatly increase the testing time, though. If it did require that, is there a way to entirely automate it, including the save / reload / fire? For my testing, I'm usually firing 2,000 shots per variation on distance and final firing accuracy... and there are a lot of variations I want to test. If you could explain more that's fine - or just let me give it a whirl and it ought to explain itself, eh? Even if it does require reloading, there may be instances where I'm checking one particular min or max sort of extreme, where even low numbers will be ok (like when we tested the floor and ceiling for psi attacks; this needed less repeats than testing within the "attack curve" itself).
I can import results data from a .CSV, if that's something you were saying.
You can email it to mikestar at speedfactory.net or upload it to SC.
Thanks for anything you can do! - MikeTheRed 07:13, 11 November 2006 (PST)
Weapon and wall strength of 1 doesn't work. Remember, the weapon can only deal a maximum of 75% of it's rated damage, so with that setup the gun'd never make a mark.
You could set the weapon strength to 2, and the wall strength to 1. That'd give you 50% accuracy. But that's just not good enough when you can record 100% of shots made so much more easily. Sure it slows things down a little, but in my view that's irrelevant. Accuracy is everything.
It's still difficult to tell whether a shot that hits the target was perfectly aimed or not. One way might be to create some special tiles using the maps in LOFTEMPS.DAT to partially mask the alien, which would prevent anything but an exact shot from reaching the target. That'd take a little while to do, so I'll leave that idea for now.
The idea of the logger is to simply record where each shot ends up. If you know where the shots were made from, then this can easily be used to work out what the angle of each shot was, with a decent (but still limited) amount of accuracy. Though there's no reason why the logger can't work out the angles and record those, too, so I'll just scribble something up tonight to spit out those with the co-ords.
And yeah, full automation is my intention. I wouldn't do it any other way. ;)
- Bomb Bloke 22:09, 11 November 2006 (PST)
Sounds great, man. Right, a weapon strength of 2 is no problem. My brain was still thinking in terms of explosion damage, laugh.
In lieu of questions, let me see what you've got... it sounds real intriguing. In the meantime - hoping for a goodie from ya - I've been working on a boundary question for which numbered tiles (and probably your util) are irrelevant. Specifically, the lowest final accuracy that guarantees a hit, vs. distance. It's almost done.
Standing by! - MikeTheRed 22:20, 11 November 2006 (PST)
Finally put the finishing touches on the thing. You can find it in the AutoHotKey thread.
What's this about lowest possible accuracy? My motto is, if there's a line of fire, an autoshot will work nine times out of ten. You're never "guaranteed" a hit, but distance doesn't ever "guarantee" a miss either.
- Bomb Bloke 19:30, 16 November 2006 (PST)
Just saw the updated thresholds. What's going on at range 1 is that the lowest FFA whose frustrum is completely intercepted by the Muton "in standard facing" is FFA 66...that is, all of the "accidental hits" are actually happening. As such, intensive testing is in order (this is the test case that gives us useful geometry information on the frustrum).
Incidentally: perhaps you should document the standard facing? [I would think that what would matter is "relative facing" and whether the distance 1 was horizontal/vertical or diagonal. 16 test cases at range 1....]
- Zaimoni 22:02, 27 November 2006 (CST)
I found it very odd that FFA 45 to 65 at Range 1, all had extremely few misses. Considerably fewer than the percent of misses that occurred, at any farther range. They did not seem to follow a decreasing pattern. I'm wondering if there was "exactly one pixel" (or whatever) somewhere odd, like right beneath the crotch or whatever, that was allowing a bullet through, ever so occasionally. And when I went to FFA 66, the cone of fire for "misses" finally was small enough to not include that one little "window" through the Muton. This could explain why it was so small, and also so invariably small across a range of FFAs.
Here's a screenshot showing my testing setup. For farther distances, my guys always move straight back. The facing and orientation for BB's fire-log map is the same. There might be testing of diagonals and other facings in the future, but for now I'm keeping it simple.
- MikeTheRed 23:36, 27 November 2006 (PST)
Yes, that should be what's happening: at range one, a lot of angles map to the same pixel after integer math.
- Zaimoni 7:54, 28 November 2006 (CST)
Link is working - shame about the page
I wanted to update to say the link to the Strategy Core thread is fine, my network card was playing up.
There is something weird about this talk page though. I can't edit the section above, and when I edit the whole page, the section above does not exist! Spike 16:15, 13 March 2009 (EDT)
Comments
Hi MTR - this is great work, I only just stumbled across it. As I noted in linking it to Weapon Analysis, it has implications for my firepower model. In my model I take 2 extreme cases: "Shock" where accuracy doesn't matter at all, and "Skirmish" where accuracy is critical. I then lamely say that real situations lie between the two. You have mapped out that ground here (and 2 years ago!).
Since when you did this research, the LOFTEMPS understanding has improved and would probably allow you to determine the actual width of a Muton, plus any "holes" (crotch or otherwise). If I was re-running this test, I would try a test at zero FFA. As per your comments, it's a way of putting a floor in, it would give the pure random element isolated from any skill element. And incidentally it would also provide a useful (?) model of reaction fire by gun toting melee aliens.
I would also shy away from testing when FFA is up near to 100%. Since by definition you can't observe greater than 100% success, you are potentially 'clipping' the top side of the (observed) function.
Excellent idea to reverse WA and FA and prove you get the same results, by the way - very rigorous.
If I had to guess how the model worked, I would guess the engine introduces a random angular error to any miss (anything failing FFA roll).
For example, your observed (and modelled) max spread of 5 at range 10, if that means max of 2.5 left or 2.5 right, another way of putting that is up to 14 degrees left or 14 degrees right, a total 28 degree cone (in the horizontal plane). Now, in the worst case, did the game engine decide to go 2.5 squares left per ten squares forward? Or did it decide to go 14 degrees left?
From what I understand of your model, you're assuming a "perpendicular" error, where there is a "left or right by up to X" error and X is a direct proportion of the distance. It should be feasible to tell if the error is angular or perpendicular. If it's an angular error, e.g. "I missed so I will now roll a random +/- 0-20 degrees left or right", then the distribution of the error left or right on the X axis will approximate a trigonometric tangent function. If the error mode is perpendicular, "I missed so I will now roll a random +/- 0 to (distance x X) left or right", then that will produce a linear distribution of error left and right.
Once the "dice are rolled" these amount to the same thing, and in the game engine once bullets are in flight they are probably implemented as X,Y,Z coordinate system vectors rather than as angles.
If you visualised this on the "wall behind the Muton", with perpendicular error the bullet holes are uniformly distributed, with angular error they are clustered around the Muton and spread further apart, the further they are away from the Muton (target).
Also if you could study the distribution of this error you could determine if it was genuinely random, or was related to (1-FFA), i.e. the "lack of accuracy" of the firer/weapon combination. In other words, do bad marksmen have wider error cones than good marksmen?
Surprisingly, from your work it also looks like the engine sometimes or always introduces a smallish error into accurate shots - enough to take 1/10th of the expected accuracy off at longer ranges. This could be a deliberate simulation of 'windage', or it could be truncation/precision error - both as suggested by NKF.
Also, as we all know from experience, there is a vertical error component as well. This might well explain some of the deviations between model and experiment. But its worth seeing if angular vs perpendicular randomness is the explanation. If you still have your raw test data we could take a look at it. Actually the worst imprecision in the data probably comes from the fact that an observed 'hit' is 'rounded' to the nearest 'integer' map square. I bet this bleeds off a lot of precision that actually exists within the game engine, and would be observable by us if we only had a way to log the pixel-path traced by the bullet, rather than just its crude endpoint. If we were going completely mad, we could use LOFTEMPS to build a sort of diffraction grating, slicing squares up into tenths or hundredths with part impenetrable wall, part air gap, and some kind of Big Red Death Tile in the next square over so you could clearly see when it passed through the air gap. But that's getting a bit mad!
Anyway, awesome work MTR. Spike 16:18, 13 March 2009 (EDT)
- I read the Strategy Core thread you linked to on the main article, and it addresses most of these suggestions and comments - and many more. Spike 17:11, 13 March 2009 (EDT)
As shown in that thread, BombBloke made an insanely cool tool for automated data collection; you let your PC run overnight while it collects data on hits and misses, noting the exact tile hit. Unfortunately as I said in my Jan 9 2007 post in that thread, I collected tons of data (36k points) which I was having trouble finding time to analyze - and then discovered I had been using an older map (without a ceiling) and for best results, should start over. Finally I found a major discrepancy between my first big set of automated data and second big set, when they should have found the same thing - something was very wrong (or misunderstood) across sessions, which threw the quality of all the data into doubt. Was I gonna have to do lots of big sessions just to figure out which of my sessions had good data and which had bad, nevermind actually getting any results?
Between these problems, I simply stopped. After you put all that time into the setup - sorry, BB!
Note that even once you have a ton of data, there is still a lot of work to be done, such as the fact that the raw data shows which tiles were hit (wall behind target, ceiling, and floor), but ultimately you want to transform that "awkward 3D data" (different tile orientations, widths, etc.) into a nice theoretical flat bulls-eye dartboard type distribution behind the target. It's hardly impossible, but it is work.
I got his tool after writing the Firing Accuracy Testing page (that's why the link to BB's tool is the last thing on the page, then I eventually dropped it as I just said), and it's certain that the results of the automated testing (which included actual tile-hit and angle data) would have made my previous work look like scribbles on a napkin. If you're interested, please go for it! Myself and probably BB will be happy to help you get the setup and Autohotkey working. Hell, AutoHotkey is really cool for almost any game - you can e.g. switch any hotkey to be anything else, even if a game doesn't allow it. I've used it just to switch key assignments on lots of games.
In retrospect I think it would be cool to try variations on the target, e.g., what if it's the smallest possible LOFTEMPS (1x1 pixel silhouette); do you still get a fixed amount of "guaranteed" hits, like I postulated? Versus other simple geometric targets like 10x10 pixels (or whatever), or the entire silhouette (what's that, 12x16 if memory serves? ... this would actually be a test that straight lines are followed and the tile directly behind the alien does not take a single hit), or the entire silhouette with a small opening, etc. etc.
Your idea of zero accuracy is also a great one. In fact there's a lot of things that could be tried. But I've become too busy with other games to sink much time into XCOM these days. Just ask Zombie... he's been waiting months for me to put in a few more hours on finishing some WORLD.DAT work (sorry, dude! I will get to it). I still don't mind talking a lot though, lol.
One other thing about this Firing Accuracy work is that I'm not good with trigonometry. I'm good with data, modelling, and probability, but not so good with angles (and 3D stuff, when relevant). Versus e.g. Zaimoni (a wiz with trig), Pi Masta (with his WORLD.DAT world map), BombBloke (as seen in that thread and e.g. his Talk page), NKF, and others. While this is ok - me, them, Zombie, Danial, we all bring skills (from the mundane to the ultra cool) that work together to make this place an amazing repository of XCOM info - it still made this particular work a little more of a challenge to me.
I see what you're saying, angular versus perpendicular. In my testing, I was actually watching the screen while I rapid-fired shots, as shown in the screenshot. Using the DOS version in DosBox, I could increase its running speed and shoot as fast as I could click (and check UNITREF.DAT later for the number of hits). I don't think it can be cranked up in the Win CE edition (but could be wrong). Anyway, in this setup, there was no way to keep track of the distribution of misses, except to make a mental note of how wide the "farthest" misses went. (Conversely, BB's automated sampling setup is a natural for capturing the actual tiles hit.)
Still, one thing is clear: whether it's angular or perpendicular, the equation also has a term for distance. Otherwise, Test 1 would have shown an ultra-simple (straight line) relationship versus spread, when I changed the distance between shooter and target.
If you want my old data, I can send it to you (4 MB .mdb, zips to 537k).
Note: You're right, normally I wouldn't want to test near 100% except I was doing something different in Test 3 - I wanted as small variability in the sampled data as possible so that any differences would be the most evident. It wasn't regular firing accuracy testing; it was seeing if the WA and SA contributions to FFA mattered. Still, it could be done at 50% FFA too, shrug.
Good to talk with you again! -MikeTheRed 21:13, 13 March 2009 (EDT)
Well I must admit I'm slightly daunted by that! I downloaded BB's fire testing tool but I haven't set it up yet - it does seem extremely cool. If you could upload your .mdb file I will take a look and see if I feel up to the task. :)
From the SC thread, it looks like a conclusion was reached that there is not a hard distinction between 'hits' and 'misses', with a hit occurring below FFA% and a 'miss' being generated above FFA% - what you called a 'deliberate miss'. That instead, you just have a single (hidden) accuracy function that generates a vector for the bullet, and that vector sometimes intersects the target - roughly (9/10)xFFA% of the time at extreme range. This sounds like a cleaner system. The one pixel LOFTEMPS test would be a good critical test of that. Actually it might have to be a 2x2 target, since with a 1 pixel target a line through the integer-rounded, computed centroid of the target might pass exactly through one of the corner vertices of the 1 pixel target - similar to the diagonal wall bug. That's testable though.
You could even test the size (and shape) of the cone angles by creating a series of concentric targets, of increasing diameter, one behind the other. You're talking zillions of test runs though, unless you just use that to verify an existing model that seems pretty solid already.
Because I can't quite believe the levels of staggering genius that would be required to concoct and tune a "single function" for hits and misses, I'm still slightly leaning toward a 'deliberate hit' vs 'deliberate miss' model, where if a shot 'rolls' below FFA% (or some lower value), it's a 'perfect' shot, and if it rolls above that %, some error is introduced (either angular error, or perpendicular error per distance). Of course, we have been surprised before about the staggering level of detail under the hood in this game (e.g. LOFTEMPS), so it's perhaps not best to underestimate the designers.
One good result in the SC thread was that you guys did find that the error cones were smaller for good marksmen. It would be interesting to investigate if this greater error was simply due to the greater range of "amount missed by" that is experienced whenever FFA is low. For example if my FFA is only 20%, I can "miss by" up to 80, whereas if FFA is 90, I can "miss by" only up to 10. But alternatively, the error angle could be a direct function of FFA (or WA, or FA): I missed, and my FFA is 20, so my error cone is (for example) 120 degrees minus my FFA. Anyway just speculation.
You are definitely good on the statistics. I misunderstood why you were testing near to 100% but re-reading your notes in more detail, I can see what a cunning statistical exercise it was. For analysis like that I would have to let Excel do the work for me, as I don't have the knowledge. (Or maybe I'll download a trial version of SPSS :) )
One point that occurred to me on the test setup methodology. You were pushing the damage up to 255 on the Laser Rifle to avoid zero damage rolls. But instead you could use an HE weapon (of power=2 perhaps) to guarantee 1-3 damage without affecting any other results. Especially since your target is suspended in mid-air.
Well I might just give this ago, starting with the "zero FFA" test case, and maybe the "2x2 pixel target" test case.
Thanks for coming back to me Mike! Spike 08:43, 14 March 2009 (EDT)
- Something's just occurred to me that might explain your unexpected result that the Average % Hit did not fall to FFA%, but fell below FFA to about 5/6ths FFA. (At FFA of 45%, average hits looked like they fell off to 39% "at infinity"). Maybe FFA is the chance to hit the square at some notional range? Maybe your target occupied about 39/45ths of the cross sectional area of the square (especially if we ignore the vertical component, which seems to be less important).
- In which case, maybe there is a "single accuracy function" that does something like this (for the horizonal plane):
My FFA is "X". Therefore, I construct a firing cone of angle "A", such that, at a notional distance "D", X% of the firing cone angle = the width of one square. (X% A = inv tan (0.5/D)). I then generate a bullet on a random angle within that firing cone.
- Actually 39/45 is exactly 5/6 - which is a suspicious number. The maximum cross section of an object in a tile is 16x24 = 384 pixels. Maybe your target had a 320 pixel cross section (5/6 x 384)? Or maybe there's a much simpler explanation, some kind of 5/6 "dice roll".
- It would be very interesting to repeat the tests against targets of different size. At the very least, you would expect the "misses that hit" to be inversely proportional to some dimension of the target - width, or cross sectional area. Or in fact, inversely proportional to the "angular" width/area that they present to the firer at that distance.
Spike 09:32, 14 March 2009 (EDT)
I, personally, am a great believer in the "angular accuracy" theory, and believe I've proved it on my talk page.
Furthermore, I do not believe the game ever decides whether a bullet will hit a target prior to the event actually happening. It determines the angle, sends the bullet on it's merry way, and whether it'll hit the target or not depends entirely on the "angular accuracy" formula and how big the target is.
I was on the way to working said formula out but I kinda got sidetracked (as I so often do) and stopped generating the numbers needed to pull it off. As it stands, it'd probably take me a week of overnight logging to get the sort of figures I reckon are required.
Though I admit part of that has to do with the hope that Seb will save me all the trouble and simply locate (& translate!) the code in the executable. I'm more about "working out the best way to do things" then actually doing them, and in this case, I honestly believe that'd be the most direct and accurate route. Call me lazy if you will. ;)
- Bomb Bloke 10:20, 15 March 2009 (EDT)