Talk:Firing Accuracy Testing

Arg, I get an error when trying to open your bb_mapedit3.zip ... is it just me? (is there a new version of zip??)

Does your Editor only edit a given savegame? (Was that just the custom uniform mod you were talking about with Zombie that mods other things?) I don't want to muck with anything but a savegame without knowing about it (thrice burnt by XcomUtil!).

By the way, I am doing Firing Accuracy testing and seeing some interesting but odd numbers. Remember that ultra cool numeric tileset you made for explosion testing? I could really use something like that, but with a twist. Particularly I need some walls (in addition to ground) that reflect damage. Just a simple north wall is enough. The point is to see the spread of misses. I seem to recall there's only so much terrain storage space... I expect a lot more hits on the walls than ground; maybe a 9:1 ratio. It's also ok if the numbers "wrap around" because this will only happen in a heavily hit area, making it clear where it transitioned. What I'm after is a "silhoutte" of misses (including ones that hit ground). Start the counting with 1 and go as far as you can. If you want more details before starting, just ask.

If you don't have the time to do it, I understand totally. I'll still have plenty of results... there just won't be as much understanding of why they are the way they are.

- MikeTheRed 17:12, 7 November 2006 (PST)

---

Most likely your download got corrupted. Try it again. I haven't spotted any updates to the ZIP format in years, though you could try an update if all else fails.

Neither the uniform mod or the editer changes anything besides save files. Both however add the files MAN_A.SPK through to MAN_N.SPK, and modify save files so that they get used.

The tileset already contains walls for you to use. To implement them, either use my editer on one of your savegames which uses the tileset, or run Diashiva's Map Editer on the MAP files. In addition, you'll find the file "desert.exe" in your terrain folder, which allows you to easily modify the armor values of the tiles.

The walls aren't numbered, however, but they don't need to be if all you're testing is where the shots go. Set the armor to 1, and if the wall gets destroyed, you know it was a hit. Also keep in mind that the chances of the ground being hit depend on how high the shooter is in relation to the target.

- Bomb Bloke 23:34, 7 November 2006 (PST)

---

Your editor opens fine now, and I see my previous download was too small... apparently I didn't notice an incomplete download (blush). Thanks for the note re: what your apps change. Almost nothing except the savegame... good deal.

What I'm asking is specifically whether you might be able to make numbered walls. Like you did for ground. So that I can see not just whether they're hit, but how often. (I meant a distribution spread, sorry.) Having both numbered walls and ground would be best. Walls only would be best, if only one can be done. But I repeat, it's not critical. Basically it will probably help in making a precise formula governing in-game accuracy... but it won't change the actual results, which are more than enough to see how it works in general. (Actual accuracy can be very different from the stated accuracy, in complex ways.)

- MikeTheRed 15:13, 8 November 2006 (PST)

---

I'm sorry, but this is one thing the game engine won't let me pull off. The reason is that when a tile is hit with a projectile, it takes a random amount of damage - 25% to 75% of the weapons capabilities.

The "armor" of each wall would need to be at most equal to that 25% mark, to ensure that each and every shot made a mark. However, if the dice roll comes out at 75%, then the tile will switch three times, not just the once - Making it impossible to get any trustworthy results after the very first shot had been made!

In case that that doesn't make any sense, I'll simplify it somewhat: At best, tile counters will go up by a random amount, from 1 to 3, with each shot. Therefore, to get any accuracy, you'd need to make a shot, then reload the map, and repeat - Making any shot counter system that counts above one, redundant.

I've crafted a suitable map, and the logger will be the work of just a few minutes. However, I'm stumped as to which format to use to save the results. Usually, I go for comma seperated data, but in this case the data is in the form of co-oridinates...

If there's some sort of 3D vector program that would be suitable to display the lines of fire, then it'd be great if I could get hold of the file format. Otherwise, I'll just dump the co-ords any old how, and write a program to show minimum/maximum/average/etc firing angles after.

- Bomb Bloke 20:42, 10 November 2006 (PST)

---

Hmm, well, I had been thinking of using weapon strength 1 and wall armor strength 1... the wall would take a hit half of the time, and I'd just take into account that I was only seeing half the hits, on average. It's much better than nothing, even if it's not a perfect counter.

What I want to see is things like, how wide the shots miss by, whether misses are equally spread from farthest left to farthest right (like, a simple random roll for the angle for misses), or whether they're concentrated near the intended target, or what. There appear to be two components to accuracy, shots that are rolled as hits, and shots that are rolled as misses; the question is the degree to which a miss roll actually hits because it's random-miss angle happens to intercept the target anyway. So the equation for predicting hit percent has at least two terms, and may need more... there may also be percent of rolled hits that actually miss, in addition to this apparent percent of rolled misses that actually hit. All of which varies by distance, of course.

For this, then, I would've liked a "wall counter" map, even if it's not perfect. By the way, I really doubt I'd need to count to more than 100 for any given tile or wall, given my setup of ammo count etc. If you can do the wall, I'd still be very interested in it.

I'm intrigued by you other suggestion, but am not sure exactly what your logger will be counting. Anything that would require me to save and reload after each shot would greatly increase the testing time, though. If it did require that, is there a way to entirely automate it, including the save / reload / fire? For my testing, I'm usually firing 2,000 shots per variation on distance and final firing accuracy... and there are a lot of variations I want to test. If you could explain more that's fine - or just let me give it a whirl and it ought to explain itself, eh? Even if it does require reloading, there may be instances where I'm checking one particular min or max sort of extreme, where even low numbers will be ok (like when we tested the floor and ceiling for psi attacks; this needed less repeats than testing within the "attack curve" itself).

I can import results data from a .CSV, if that's something you were saying.

You can email it to mikestar at speedfactory.net or upload it to SC.

Thanks for anything you can do! - MikeTheRed 07:13, 11 November 2006 (PST)

---

Weapon and wall strength of 1 doesn't work. Remember, the weapon can only deal a maximum of 75% of it's rated damage, so with that setup the gun'd never make a mark.

You could set the weapon strength to 2, and the wall strength to 1. That'd give you 50% accuracy. But that's just not good enough when you can record 100% of shots made so much more easily. Sure it slows things down a little, but in my view that's irrelevant. Accuracy is everything.

It's still difficult to tell whether a shot that hits the target was perfectly aimed or not. One way might be to create some special tiles using the maps in LOFTEMPS.DAT to partially mask the alien, which would prevent anything but an exact shot from reaching the target. That'd take a little while to do, so I'll leave that idea for now.

The idea of the logger is to simply record where each shot ends up. If you know where the shots were made from, then this can easily be used to work out what the angle of each shot was, with a decent (but still limited) amount of accuracy. Though there's no reason why the logger can't work out the angles and record those, too, so I'll just scribble something up tonight to spit out those with the co-ords.

And yeah, full automation is my intention. I wouldn't do it any other way. ;)

- Bomb Bloke 22:09, 11 November 2006 (PST)

---

Sounds great, man. Right, a weapon strength of 2 is no problem. My brain was still thinking in terms of explosion damage, laugh.

In lieu of questions, let me see what you've got... it sounds real intriguing. In the meantime - hoping for a goodie from ya - I've been working on a boundary question for which numbered tiles (and probably your util) are irrelevant. Specifically, the lowest final accuracy that guarantees a hit, vs. distance. It's almost done.

Standing by! - MikeTheRed 22:20, 11 November 2006 (PST)

---

Finally put the finishing touches on the thing. You can find it in the AutoHotKey thread.

What's this about lowest possible accuracy? My motto is, if there's a line of fire, an autoshot will work nine times out of ten. You're never "guaranteed" a hit, but distance doesn't ever "guarantee" a miss either.

- Bomb Bloke 19:30, 16 November 2006 (PST)

---

Just saw the updated thresholds. What's going on at range 1 is that the lowest FFA whose frustrum is completely intercepted by the Muton "in standard facing" is FFA 66...that is, all of the "accidental hits" are actually happening. As such, intensive testing is in order (this is the test case that gives us useful geometry information on the frustrum).

Incidentally: perhaps you should document the standard facing? [I would think that what would matter is "relative facing" and whether the distance 1 was horizontal/vertical or diagonal. 16 test cases at range 1....]

- Zaimoni 22:02, 27 November 2006 (CST)

---

I found it very odd that FFA 45 to 65 at Range 1, all had extremely few misses. Considerably fewer than the percent of misses that occurred, at any farther range. They did not seem to follow a decreasing pattern. I'm wondering if there was "exactly one pixel" (or whatever) somewhere odd, like right beneath the crotch or whatever, that was allowing a bullet through, ever so occasionally. And when I went to FFA 66, the cone of fire for "misses" finally was small enough to not include that one little "window" through the Muton. This could explain why it was so small, and also so invariably small across a range of FFAs.

Here's a screenshot showing my testing setup. For farther distances, my guys always move straight back. The facing and orientation for BB's fire-log map is the same. There might be testing of diagonals and other facings in the future, but for now I'm keeping it simple.

- MikeTheRed 23:36, 27 November 2006 (PST)

Yes, that should be what's happening: at range one, a lot of angles map to the same pixel after integer math.

- Zaimoni 7:54, 28 November 2006 (CST)

Link is working - shame about the page

I wanted to update to say the link to the Strategy Core thread is fine, my network card was playing up.

There is something weird about this talk page though. I can't edit the section above, and when I edit the whole page, the section above does not exist! Spike 16:15, 13 March 2009 (EDT)

Comments

Hi MTR - this is great work, I only just stumbled across it. As I noted in linking it to Weapon Analysis, it has implications for my firepower model. In my model I take 2 extreme cases: "Shock" where accuracy doesn't matter at all, and "Skirmish" where accuracy is critical. I then lamely say that real situations lie between the two. You have mapped out that ground here (and 2 years ago!).

Since when you did this research, the LOFTEMPS understanding has improved and would probably allow you to determine the actual width of a Muton, plus any "holes" (crotch or otherwise). If I was re-running this test, I would try a test at zero FFA. As per your comments, it's a way of putting a floor in, it would give the pure random element isolated from any skill element. And incidentally it would also provide a useful (?) model of reaction fire by gun toting melee aliens.

I would also shy away from testing when FFA is up near to 100%. Since by definition you can't observe greater than 100% success, you are potentially 'clipping' the top side of the (observed) function.

Excellent idea to reverse WA and FA and prove you get the same results, by the way - very rigorous.

If I had to guess how the model worked, I would guess the engine introduces a random angular error to any miss (anything failing FFA roll).

For example, your observed (and modelled) max spread of 5 at range 10, if that means max of 2.5 left or 2.5 right, another way of putting that is up to 14 degrees left or 14 degrees right, a total 28 degree cone (in the horizontal plane). Now, in the worst case, did the game engine decide to go 2.5 squares left per ten squares forward? Or did it decide to go 14 degrees left?

From what I understand of your model, you're assuming a "perpendicular" error, where there is a "left or right by up to X" error and X is a direct proportion of the distance. It should be feasible to tell if the error is angular or perpendicular. If it's an angular error, e.g. "I missed so I will now roll a random +/- 0-20 degrees left or right", then the distribution of the error left or right on the X axis will approximate a trigonometric tangent function. If the error mode is perpendicular, "I missed so I will now roll a random +/- 0 to (distance x X) left or right", then that will produce a linear distribution of error left and right.

Once the "dice are rolled" these amount to the same thing, and in the game engine once bullets are in flight they are probably implemented as X,Y,Z coordinate system vectors rather than as angles.

If you visualised this on the "wall behind the Muton", with perpendicular error the bullet holes are uniformly distributed, with angular error they are clustered around the Muton and spread further apart, the further they are away from the Muton (target).

Also if you could study the distribution of this error you could determine if it was genuinely random, or was related to (1-FFA), i.e. the "lack of accuracy" of the firer/weapon combination. In other words, do bad marksmen have wider error cones than good marksmen?

Surprisingly, from your work it also looks like the engine sometimes or always introduces a smallish error into accurate shots - enough to take 1/10th of the expected accuracy off at longer ranges. This could be a deliberate simulation of 'windage', or it could be truncation/precision error - both as suggested by NKF.

Also, as we all know from experience, there is a vertical error component as well. This might well explain some of the deviations between model and experiment. But its worth seeing if angular vs perpendicular randomness is the explanation. If you still have your raw test data we could take a look at it. Actually the worst imprecision in the data probably comes from the fact that an observed 'hit' is 'rounded' to the nearest 'integer' map square. I bet this bleeds off a lot of precision that actually exists within the game engine, and would be observable by us if we only had a way to log the pixel-path traced by the bullet, rather than just its crude endpoint. If we were going completely mad, we could use LOFTEMPS to build a sort of diffraction grating, slicing squares up into tenths or hundredths with part impenetrable wall, part air gap, and some kind of Big Red Death Tile in the next square over so you could clearly see when it passed through the air gap. But that's getting a bit mad!

Anyway, awesome work MTR. Spike 16:18, 13 March 2009 (EDT)