[Search] Search   [Recent Topics] Recent   [Hottest Topics] Hottest   [Login] Login
Proposal to change the way maps are ranked
Forum Index » New Feature Request
[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline

[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline
This is an outcome of a chat discussing ways to improve the rating of user generated maps.

Current Map Rating System
The current process is every player is given an option to up or down vote the race match up at the end of every match. A map's vote score is determined by (up / (up + down) ). There is a weighting applied to the vote so that players with higher rank have more influence over the rating than lower rank players. The vote is further weighted such that vote down on win and vote up on loss are more significant that vote up on win, vote down on loss.

The problem
There are no maps that hold (let alone gain) vote score over the long term. Recently it was discovered that the ranked map pool was down to only 12 maps and a change was made so that rather than only allowing ranked maps with a score of >70% up vote, we always define the top 100 maps ordered by vote score are ranked. The effect of this change is there are many maps coming back for a brief while before being sent back out of ranked, but it does not solve the problem that in all cases maps are still voted down often enough to remove them from the ranked pool.

Reasoning
Map makers generally agree that the aim of any ranked map is to be balanced for all races in either position for all players, such that 2 roughly similar ranked players have an equal chance of victory and it is their skill on that map that makes the difference and not the map configuration itself. The best reason we can come up with as to why all maps eventually get voted down is because it is not possible to create such a map. Different players with different play styles at different levels will have vastly different experiences on a given map. High ranked players may feel a map is well balanced for all races but if lower ranked players cannot see any way to win with a particular race they will quickly down vote it. Equally maps that seem well balanced to lower ranked players can have fixed sequences that are completely indefensible that seem obvious to high ranked players, again the map is voted down. Then some people just get bored if they see the same map more than a few times and others don't like certain styles of map e.g. using blank tiles. All result in more down votes.

Android 9. Samsung Galaxy A50
[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline

[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline
A Proposed Solution
Maps should use the objective race match up score to determine if they are ranked. For each map the races and result of every game is recorded. This information can be used to determine if a map is balanced . There are 9 possible match ups for a map.

For each cell (1 - 9) in the above table there are 2 counters, P1 wins, P2 wins. At the end of each match the appropriate cell win value is updated. In the case of a draw i.e. the AI plays 35 rounds on small, 50 medium etc then both values are incremented. This allows us to create a percentage chance of winning (or losing) for any race match up. i.e. P1 Win% = (100 * P1) / (P1 + P2). When the percentage chance of winning for any single match up is greater than say 65% then the map becomes unranked. Hopefully all of this is already implementable using the data that is already stored.

* Increasing the win amount by more or less depending on whether the result was expected i.e. 1900 vs 1800 there is a expectation the 1900 ranked player should win so if they don't the win count for the 1800 player could be increased by a larger amount to show that it is easier to win from that location.
* Higher ranked players should also have more influence over the win/loss ratio than lower ranks.
* As the total number of games increases the threshold at which the map becomes unranked could be reduced.

Displaying the Ranked Status
The ranked status would need to be displayed to show how balanced a map is. The simplest option seems to be to show the worst race match up, as this is the match up that will cause the map to go unranked. Colouring the percentage green through orange to red as the highest rating reaches the threshold would help visualise the unfairness to the map. e.g.

Win Ratio = 62% S vs 38% K
Win Ratio = 59% T vs 41% S
Win Ratio = 47% K vs 53% K

Clicking on the the ratio should bring up the full list of 9 sets of ratios ordered from worst to best.

Blending in the popular vote
The current voting system could still be in place as it gives a popular oppinion of a map. Using this vote to order the rated maps and decide which maps get assigned to new users would still be useful, offer prizes if enough votes are reached, create all time favourites/most hated lists, etc

This message was edited 4 times. Last update was at Dec 15, 2019 23:44


Android 9. Samsung Galaxy A50
[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline

[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline
Getting the map into the ranked pool
Currently a map becomes ranked when it reaches 25 up votes and maintains a 70% up vote ratio. As the voting would be removed a different method is needed.
One of the issues with using the win/loss for each match up is it needs lots of games before any initial ratios make sense minimum 100. This is only realistic using some automated process.

If the map editor was updated so that an AI could play another AI then a record of games for a map could be created before upload. The AI's could play each other without any UI display and return a set of results. If a map is roughly balanced then the 3 current AIs (even though they play differently and badly) would give an initial set of results for balance. Maps are still pushed to the server as they do currently but they are not available for ranked unless they have cleared the minimum number of runs. Once the local test runs have reached the required number of runs and have maintained balance the map gets flagged so the server knows it can then runs its own tests. The server simply has to run automated AI vs AI matches on a map that has less than 100 (or what ever minimum). If the map is balanced then after that amount the map would be ranked if not it would remain unranked.

Summary
Changing to win/loss ratio should mean that truely balanced maps are kept in the system for all to enjoy and truely imbalanced maps are removed. It would also clearly identify to map makers the particular match ups that are not working when maps are imbalanced. The automated local play would also give a way for map makers to test that their new map is actually balanced and if not to identify where the imbalance is.

Hopefully in time there would be a large list of well balanced maps with many plays that all can enjoy.

Android 9. Samsung Galaxy A50
[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline

[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline
I'm going to try and test the AI match up theory by taking some maps converting then to challenge maps and adding 1 unit for player 1 then making the original p1, p2 and p2, p3. Then i can start 90 games and set p2 and p3 as ai and watch them fight, then record the winner. It'll take a while.

Notes. I've spotted a potential issue with my manual test. Because it is always the same ai playing itself we aren't getting as much variety in the play styles. Also if the AI play style is bad at playing Titans (aren't we all?) then it will always lose with them until the map is changed so heavily weighted in their favour that any human player will find it unbalanced. Is this just highlighting that map balance is variable as player skill increases? More data on more maps is needed, sadly lots

Notes. I have been playing every match to the bitter end, ignoring the draw criteria, which I'd forgotten in the initial write up. Lower draw criteria means more draws, which means the total increases for a match up and reduces the bias that an AI might have when playing. It essentially says if the AI can play for x rounds then it's probably balanced enough.

Notes. 90 matches i.e. 10 per match up is barely enough to give unbiased stats. However we can save time running more by focusing on the non balanced areas. I.e. if 1 matchup is showing 8,2 then we run another 10 on that. If it's 5,5 then we're happy it's balanced enough, no need to run more. Repeat for any unbalanced after 20 runs. If it's still got strong bias after 30 runs then it's not the rng there's definitely some imbalance.

Notes. Moving P1 to P2, P2 to P3 is not quite the same as P1vsP2 because P1 always get the initial reduced credits. For most maps that doesn't matter as they have an initial phase of capping bases nearby but it is subtly different.

This message was edited 5 times. Last update was at Dec 18, 2019 15:13


Android 9. Samsung Galaxy A50
[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline

[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline
Map: Deadmonk AI Test
Mix of AI modes. 11-20 TvS used #AI1



After 90 games.
T 0% vs S 100%
S 90% vs T 10%
K 90% vs K 10%

K 80% vs T 20%
T 70% vs T 30%
K 70% vs S 30%

S 60% vs K 40%
S 40% vs S 60%
T 40% vs K 60%

This message was edited 94 times. Last update was at Dec 18, 2019 15:10


Android 9. Samsung Galaxy A50
フ乇千千卩卄79

Messages: 4,
Joined: Apr 21, 2018,
Offline

フ乇千千卩卄79

Messages: 4,
Joined: Apr 21, 2018,
Offline
Just to start this off, I'll give a more in depth response once I process this. I belive ai does already play in the background. If u check the game invites alot of them are ai games. At least it used to be not sure if that changes
[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline

[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline
AI does play on the device and in the background, but there must always be a player. This automated testing processes would require AI vs AI without a player. What it really needs to do is run the whole set of 90+ match ups then give you a report like the cells above, or just in a list. Given the power of your average mobile it will likely take less than 10 minutes. There would be no record of the match, just the result. To see the imbalance in the AI game you'd need to start a single game and be able to just observe, also something you can't do now.

Android 9. Samsung Galaxy A50
[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline

[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline
Map: Joy division AI Test
Note: Only using #AI1

After 90 games.
T 100% vs S 0%
S 0% vs T 100%
K 100% vs S 0%
K 90% vs T 10%

K 20% vs K 80%

S 42% vs K 58%
S 60% vs S 40%
T 60% vs T 40%
T 50% vs K 50%

This message was edited 64 times. Last update was at Dec 20, 2019 14:28


Android 9. Samsung Galaxy A50
[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline

[Avatar]
wookieontheweb

Messages: 485,
Joined: Jan 27, 2016,
Location: Southampton, UK
Offline
Championship August 2019
Results from round 1, removed players ignored
Map: War on a beach
Got to: b

TBD
TBD

TBD

This message was edited 9 times. Last update was at Jan 04, 2020 20:22


Android 9. Samsung Galaxy A50
Forum Index » New Feature Request
Powered by JForum 2.1.9 © - 2020-04-14 v124 - UniWar website