ACTUV Scoring Structure

[Gargamel]

Originally Posted by Molon Labe

I haven't tested for it, but I'd have to imagine HF contact is also per 5 minutes. Or any other sensor for that matter. As long as the light is still flashing, you're getting the points (or if the light would be flashing, but is steady because you hold contact on another sensor).

That's probably a good hypothesis too. The bonus for HF is that it's "passive" (compared to the MF). So there's no penalty for using it.

[Gargamel]

Doing some work on a sprint and drift strategy, I came up with this interesting chart.



Assuming you have maximum sensor contact on a target (Sensor angle was not considered), it gives you how many points, based on our recent 5 minute findings (and not the written scoring rules), you will earn in the next five minutes given your current speed.

One major note, We have shown that MF scoring continues into the red zone, but that gets really complicated to model in this simple chart, so it assumes that when your speeding, you have lost contact.

Green Indicates maximum scoring with both (or only) sensor, yellow is one sensor, and Red is no sensors.


Couple quick observations from this chart:

1) Once a Gator or Triton locks onto the SSK, it should be practically impossible to shake it off. The maximum sensor speed is greater than the max SSK speed.

2) The Remora is a true sprint and drift boat for intial acquisition, but once locked, it will be difficult to shake. Even if the SSK goes Flank, the Remora can chug along at 7-8 knots tracking it until it too can sprint to keep up.

3) For the Remora and Seahorse, it is imperative to regain HF ASAP, as they cannot be competitive for long with only MF.

4) The triton should be the hands down winner here, from this chart at least, as it can score max points at any speed the SSK can go. But looking at the sensor profile, it requires a deft hand to keep it locked down the whole time.


EDIT: PIC Fixed, had wrong Triton Sensor speeds input. math was right, I was just using bad data. chart is now fixed.

[Gargamel]

So, taking the above data, I compared Speed vs Range and it's effects on the score per 5 minutes for the Seahorse Platform. I will also do Triton and Remora, but I won't do Gator and Shark. Gator is either in the zone or not, and Shark's ranges depend on which way your pointing so it won't be accurate.

And while some may find this chart very intuitive, and ask why we even need it, I did it just to do it. Some may find it useful as It helps them visual the scoring 'bubble' each platform can operate in.

And to be honest, it was kinda fun, trying to work out the following formula was interesting. This had to go in each cell you see:

=IF(CP$2<=$B$33; IF($A14 <= $B$32;1500-(30/(($B$29/$A14)^2)); IF($A14 <=$B$30 ; 500-(30/(($B$29/$A14)^2)); -30/(($B$29/$A14)^2) ));IF(CP$2 <=$B$31;IF($A14 <= $B$30 ; 500-(30/(($B$29/$A14)^2)); -30/(($B$29/$A14)^2) );-30/(($B$29/$A14)^2)))

So for seahorse we get:



Each box is 50 yds, upto 7500 yds out. Sorry bout the hash marks, but when you shrink a number to less than it can show, you get a #, hence why it looks darkened.



Please comment on these, if anybody wants, I can email them the spreadsheet I've been playing with so they can mess around with my numbers.

[Molon Labe]

Gargamel, I think you're right about MF, IR, EO, and Radar not stacking. See this from the debrief file:

Goals
Critical
ACTUV contact on SSK for 30min,Complete,0
Non-Critical
HF contact - 1 minute,Complete,14700
MF/RDR/EO contact - 1 minute,Complete,12000
MF Active Bonus,Complete,986
Fuel Bonus,Complete,918

What we see here is that the scoring triggers are Event Triggers that fire every minute they return True, and that MF, Radar, and EO are all part of the same trigger.

[Gargamel]

Originally Posted by Molon Labe

Gargamel, I think you're right about MF, IR, EO, and Radar not stacking. See this from the debrief file:



What we see here is that the scoring triggers are Event Triggers that fire every minute they return True, and that MF, Radar, and EO are all part of the same trigger.

Hmm.. then yeah... forcing them to the surface is actually a disadvantage to you, as you cn no longer get the stacked HF + MF score, but just the MF/IR/EO/RDR score.

Not sure how forcing them to the surface is an option anyways

[Molon Labe]

Well, I think I've pretty much done as well as I'm going to do on all of these (except Gator, I've only played it once and I'm not going back). And it's left me a bit disappointed with the scoring structure.

As was pointed out earlier in this thread, the points given for maintaining contact on both HF and MF are so high in comparison to the fuel and pinging penalties that the decision-making "flowchart" is very straightforward. If you don't have contact on both sensors, getting contact is always your highest priority (other than avoiding proximity to surface traffic), so you never need to concern yourself with speed settings or MF sonar use until you have continuous contact on both sensors. From a gaming point of view, the lack of tradeoffs make it rather uninteresting. From a tactical research point of view, it makes me wonder what DARPA needs us for.... why use crowdsourcing to discover "optimal" tactics when the optimal approach is so obvious?

[dd149]

I can only fully share your opinion, this seems to be a big joke, the tactics are extremely basic and do no seem to offer any challenge in tactics creativity, One can only wonder why DARPA thinks that it could benefit from crowd resources on that aspect. Or is it a creative way to spy on the Subsim community for some reason? In any case, any simmer with some experience of LWAMI or RA modified DW game will not spend much time on ACTUV.

[Molon Labe]

Well, I've spent quite a few hours with it already, but I am at the point where I feel like DARPA needs to give us some sort of information or feedback (or SCS needs to change the missions or scoring structure) in order for there to be a point to go any further.

[TerminatorSub]

Originally Posted by Molon Labe

Well, I think I've pretty much done as well as I'm going to do on all of these (except Gator, I've only played it once and I'm not going back). And it's left me a bit disappointed with the scoring structure.

As was pointed out earlier in this thread, the points given for maintaining contact on both HF and MF are so high in comparison to the fuel and pinging penalties that the decision-making "flowchart" is very straightforward. If you don't have contact on both sensors, getting contact is always your highest priority (other than avoiding proximity to surface traffic), so you never need to concern yourself with speed settings or MF sonar use until you have continuous contact on both sensors. From a gaming point of view, the lack of tradeoffs make it rather uninteresting. From a tactical research point of view, it makes me wonder what DARPA needs us for.... why use crowdsourcing to discover "optimal" tactics when the optimal approach is so obvious?

You are an expert. I think they also look for newbies like me to add noise to the results.

I Googled some info on optimal submarine search patterns and chose the following papers:

faculty.nps.edu/awashburn/docs/MORV6N4.pdf
c3uv.berkeley.edu/papers/Mcgee_acc06.pdf
http://www.wired.com/wiredscience/tag/levy-flight/

It seems the choices to sweep an area are spiral in, zig-zag, or random. I tried these different approaches.

In trailing the target, one can be aggressive (perhaps too much) or lay back a bit. In some of the platforms and scenarios, it is hard to get close enough to acquire a HF lock. That blind spot in the Triton that you mentioned in another post, and the speed limited HF in some cases are factors.

Theory is only good for 5 minutes. Different players will react to the sub throwing decoys and the surface vessel attack in unique ways.

I am wondering if the score results will be analyzed with statistics, or if they will look at each result for fastest target acquisition, closest approach of the surface attack vessel, and shortest time out of contact with the SSK while avoiding said surface attacker?

I also wonder if there will be a second version with more scenarios. The SSK hasn't thrown a torpedo yet, but one should be expected.

[Molon Labe]

Originally Posted by TerminatorSub

I Googled some info on optimal submarine search patterns and chose the following papers:

It seems the choices to sweep an area are spiral in, zig-zag, or random. I tried these different approaches.

That's really outside the scope of the ACTUV. The name of the game here is continuous trail. If you need to resort to area searches, you've already "lost." (Mission 1 notwithstanding)

In trailing the target, one can be aggressive (perhaps too much) or lay back a bit. In some of the platforms and scenarios, it is hard to get close enough to acquire a HF lock. That blind spot in the Triton that you mentioned in another post, and the speed limited HF in some cases are factors.

In the real life operations of the ACTUV, this may very well be true. And that's part of the frustration I'm expressing. I really can't fathom why SCS is effectively telling us that it's better to burn tons of gas sprinting around to maintain 300yd range on a contact that's constantly making radical course changes at 20 knots, backing off to reestablish MF contact once every 5 minutes, and sprinting like a madman to get HF back, when you could sit back at 1000yd maintaining MF contact running the diesels at 20 knots or less... probably much less if the target is maneuvering.

But the way this sim is set up, maintaining both MF and HF contact at all times is the tactically correct move, regardless of fuel consumption. And if you absolutely have to lose contact on one (which is how I feel about Triton mission 3, if you're interested), be aggressive and stay with the HF, because you're penalized 50% more for losing HF contact compared to MF.

[StevenLohr]

Molon Labe-First of all, congrtulations. Your scores are awesome. IRT your frustration: While I can't speak for DARPA, I suspect that they are trying to see what solutions that the "experts" (i.e., the subsim crowd) come up with IRT this problem. Over the past 60 years, Operations Analysis (OA) has established the "textbook" solution to the particular trailing problems we are simulating. However, we are being asked to act as a "Red Team" or an alternative analysis group. By evaluating our tactics, DARPA can verify the OA solution.
Additionally, the scenarios we are being given, while simple, are dynamic. The presence of merchant traffic, false contacts and Kamikaze merchants adds a dynamic that OA would have trouble quantifying, but the trained human brain can deal with fairly easily. How we deal with these problems in a dynamic environment gives the programers some insight into how to teach the ACTUV to act. Remember the original "Star Trek" episode where the computer scientist integrated some of his brain onto the computer? As a group, we are kind of doing the same the same thing, helping to train the ACTUV computer to act/react in a dynamic environment.
As an aside, I'm pleased that DARPA is thinking "outside the box" in this endeavor. It is an excellent idea to crowd-sourcing this portion. I would hope that they continue the experiment.

[Gargamel]

I think Steve has it right. IT's not about the tracking methods, as those are pretty cut and dry. Establish contact, don't lose it and get as good a contact as possible.

I think they are trying to see how the drone should react to other factors that aren't directly scored ingame. IE, avoiding merchant traffic. The higher scoring results will have successfully avoided traffic to not get the penalty, and they are trying to see how the better drivers avoid the contact while maintaining optimal contact. If the drone sees the target going for a back scratch manuever on a merchy, how should it react to stay on target and not collide?

What I still don't get though, is the array of sensors and platforms we're being given to work with. Are they seriously using our results to help decide which design bid to accept? Some of the designs are useless at the targets top speed, so throw that out. And then why can one design have uber-sensors, while others have them seemingly pointed in the wrong direction? Why can't they mix and match? If it's a power/size issue, then make the damn thing a bit bigger. I bet some solar panels, wind, and wave generators would work nicely on this thing, etc etc. If they have intentionally designed flaws into the drones to test our tactics, then why the redundancy?

Anyways, With my internet being as it is right now, I haven't been able to submit my results (different machine). I'm just glad I was able to contribute some statistical analysis to the project.

[Molon Labe]

Originally Posted by Gargamel

What I still don't get though, is the array of sensors and platforms we're being given to work with. Are they seriously using our results to help decide which design bid to accept? Some of the designs are useless at the targets top speed, so throw that out. And then why can one design have uber-sensors, while others have them seemingly pointed in the wrong direction? Why can't they mix and match? If it's a power/size issue, then make the damn thing a bit bigger. I bet some solar panels, wind, and wave generators would work nicely on this thing, etc etc. If they have intentionally designed flaws into the drones to test our tactics, then why the redundancy?

I've been wondering about that too. If Gator is fictitious, then I'd say the Remora and Shark designs are both very good. I like Shark a little bit more because of the higher MF speed and longer HF range. It's sonar is OP though, at least in the DW acoustic model anything beyond 4nmi is a waste. Seahorse is nice, but I'd rather have the speed of the others than omnidirectional MF sonar. Seahorse's best feature might be the elevation on the MF--although with its long-range HF sonar it's really necessary. Triton blows. It would rock, though, if it traded MF elevation with the Seahorse's. Short HF range should be paired with high elevation MF.