Creating the ultimate tier list

FluffyQuack

FluffyQuack

[14] Master
There's a lot of people putting a lot of thought into tier lists for SC games. Should I play as Dampierre!? Or is Yoda a top tier character I should play for? Well, fret not, I will be able to answer those questions as I am creating the one thing no one has ever asked for: a tier list based on AI vs AI matches!

I created a trainer which automatically tracks the results from matches in Battle Theater, and I'm streaming the process of creating the tier list for Soul Calibur 2.

So if anyone has literally nothing to do at all you can check out the automated building of the tier list here: https://www.twitch.tv/fluffyquack
 
After monitoring 2626 matches over 181 hours, scientific science of science has concluded!

01: Sophitia
Matches: 242
Wins/losses: 172/70
Win rate: 71.07%
Best matchup: Yoshimitsu (Matches: 7, win rate: 100.00%)
Worst matchup: Voldo (Matches: 14, win rate: 50.00%)

02: Voldo
Matches: 262
Wins/losses: 185/77
Win rate: 70.61%
Best matchup: Ivy (Matches: 10, win rate: 100.00%)
Worst matchup: Seong Mi-na (Matches: 19, win rate: 47.37%)

03: Cervantes
Matches: 269
Wins/losses: 171/98
Win rate: 63.57%
Best matchup: Yoshimitsu (Matches: 18, win rate: 88.89%)
Worst matchup: Nightmare (Matches: 18, win rate: 33.33%)

04: Nightmare
Matches: 247
Wins/losses: 156/91
Win rate: 63.16%
Best matchup: Yoshimitsu (Matches: 7, win rate: 100.00%)
Worst matchup: Yun-seong (Matches: 11, win rate: 36.36%)

05: Cassandra
Matches: 242
Wins/losses: 146/96
Win rate: 60.33%
Best matchup: Yoshimitsu (Matches: 13, win rate: 84.62%)
Worst matchup: Cervantes (Matches: 15, win rate: 26.67%)

06: Seong Mi-na
Matches: 256
Wins/losses: 148/108
Win rate: 57.81%
Best matchup: Maxi (Matches: 5, win rate: 100.00%)
Worst matchup: Sophitia (Matches: 15, win rate: 26.67%)

07: Necrid
Matches: 250
Wins/losses: 139/111
Win rate: 55.60%
Best matchup: Link (Matches: 14, win rate: 100.00%)
Worst matchup: Nightmare (Matches: 9, win rate: 11.11%)

08: Kilik
Matches: 241
Wins/losses: 130/111
Win rate: 53.94%
Best matchup: Astaroth (Matches: 8, win rate: 87.50%)
Worst matchup: Cassandra (Matches: 20, win rate: 30.00%)

09: Taki
Matches: 269
Wins/losses: 131/138
Win rate: 48.70%
Best matchup: Yoshimitsu (Matches: 20, win rate: 80.00%)
Worst matchup: Cervantes (Matches: 13, win rate: 15.38%)

10: Yun-seong
Matches: 248
Wins/losses: 118/130
Win rate: 47.58%
Best matchup: Yoshimitsu (Matches: 15, win rate: 73.33%)
Worst matchup: Sophitia (Matches: 18, win rate: 11.11%)

11: Ivy
Matches: 241
Wins/losses: 114/127
Win rate: 47.30%
Best matchup: Maxi (Matches: 11, win rate: 81.82%)
Worst matchup: Voldo (Matches: 10, win rate: 0.00%)

12: Mitsurugi
Matches: 251
Wins/losses: 118/133
Win rate: 47.01%
Best matchup: Yun-seong (Matches: 14, win rate: 64.29%)
Worst matchup: Seong Mi-na (Matches: 18, win rate: 22.22%)

13: Astaroth
Matches: 252
Wins/losses: 118/134
Win rate: 46.83%
Best matchup: Yoshimitsu (Matches: 7, win rate: 85.71%)
Worst matchup: Voldo (Matches: 11, win rate: 0.00%)

14: Talim
Matches: 248
Wins/losses: 115/133
Win rate: 46.37%
Best matchup: Yoshimitsu (Matches: 26, win rate: 76.92%)
Worst matchup: Seong Mi-na (Matches: 12, win rate: 16.67%)

15: Maxi
Matches: 231
Wins/losses: 86/145
Win rate: 37.23%
Best matchup: Yoshimitsu (Matches: 19, win rate: 68.42%)
Worst matchup: Seong Mi-na (Matches: 5, win rate: 0.00%)

16: Xianghua
Matches: 259
Wins/losses: 94/165
Win rate: 36.29%
Best matchup: Raphael (Matches: 13, win rate: 61.54%)
Worst matchup: Voldo (Matches: 18, win rate: 11.11%)

17: Link
Matches: 234
Wins/losses: 80/154
Win rate: 34.19%
Best matchup: Maxi (Matches: 15, win rate: 73.33%)
Worst matchup: Necrid (Matches: 14, win rate: 0.00%)

18: Raphael
Matches: 250
Wins/losses: 80/170
Win rate: 32.00%
Best matchup: Link (Matches: 16, win rate: 75.00%)
Worst matchup: Voldo (Matches: 22, win rate: 13.64%)

19: Yoshimitsu
Matches: 250
Wins/losses: 70/180
Win rate: 28.00%
Best matchup: Raphael (Matches: 7, win rate: 71.43%)
Worst matchup: Sophitia (Matches: 7, win rate: 0.00%)

Notes:
- Version used was European Gamecube version running on Dolphin 5.0.
- 5 rounds to win for each match.
- Mimic characters ignored.
- Mirror matches ignored.
- Assassin, Berserker, and Lizardman ignored because the AI never chooses them in Battle Theatre mode.
- All bonus weapons locked so AI would only choose standard weapons.

But the biggest question might be: Why did I do this mad scientist experiment to begin with!?

Well, I kinda just wanted an excuse to stream Soul Calibur 2 for a week straight.

 
I guess they really struggled with Yoshi's AI, huh?
It would be interesting to compare this to an actual tier list just to see which AIs managed to get the most out of the character's potential...
 
Dunno which surprises me more, thant Seong Mi-Na is as high as she is on this list or that she's higher than Kilik (all this considering her history of playing second fiddle to the latter). This is interesting information though, and I'd like to see how this experiment can be replicated on later games in the series.
 
Fatal_Exodus said:
Dunno which surprises me more, thant Seong Mi-Na is as high as she is on this list or that she's higher than Kilik (all this considering her history of playing second fiddle to the latter). This is interesting information though, and I'd like to see how this experiment can be replicated on later games in the series.
Click to expand...
I'm gonna try to do the same thing with SC3. One extra thing I had to do with SC2 was make sure all extra weapons are locked, and I'm trying to do the same thing with SC3 but it's proving to be a bit more difficult.

I think it would be fun to do the same with SC4 and SC5 but the big problem there is that my PC isn't good enough to emulate them at full speed.
 
Now doing science with Soul Calibur 3: https://www.twitch.tv/fluffyquack

I did some brief tests with SC4, and I probably won't be able to do anything with that. First of all, I haven't been able to get the AI to fight itself, and there's the bigger problem I can't get close to emulating it at full speed.
 
The problem with allowing AI to determine placement on a tier list is no matter how accurate and without error the system may be with inputs...that would still have no bearing realistically on a pvp experience

Tbh, AI does lots of dumb shit that doesn't work against human players. Difficulty at best is still reduced to "counter" input reads
 
Fatal_Exodus said:
Dunno which surprises me more, thant Seong Mi-Na is as high as she is on this list or that she's higher than Kilik (all this considering her history of playing second fiddle to the latter). This is interesting information though, and I'd like to see how this experiment can be replicated on later games in the series.
Click to expand...
I'm surprised Kilik's that high on this list, to me he's actually one of the worst characters in the game, he's my main, so I can really feel the nerfs even though I'm better with him than most of the cast anyways, Seong Mi-na always felt like the better character in 2, it's the only game where she is better than Kilik I believe.
 
Science has reached its scientific conclusion yet again!

After watching the AI fight for quite a long time, I'm really curious to see what improvements they've made in SC6. The AI in SC3 is definitely an improvement from SC2 (though most of the improvements seem to be only about defense: the AI loves getting into GI standoffs and it loves blocking or evading attacks but not attacking while opponent is attacking), but there's so many things they could have done it to make it fight in a more human-like manner.

Here's 24 hours of AI vs AI battles: (in poor quality until youtube finishes processing the video)

Here's some notes on how I did the science:
- US release of SC3 using PCSX2 emulator (version 1.4.0).
- Maximum AI difficulty.
- Bonus characters included.
- Mimic character ignored.
- Mirror matches ignored (though Battle Theatre avoids these anyway).
- All non-standard weapons locked so AI wouldn't ever use them (this required some memory editing magic as this state is impossible via normal play).
- 3 matches to win for each match (a little bit of memory editing magic required for this as well since the game doesn't let you customize quantity of rounds for Battle Theatre).

And the tier list itself (7383 matches monitored):

01: Xianghua
Matches: 349
Wins/losses: 297/52
Win rate: 85.10%
Best matchup: Maxi (Matches: 8, win rate: 100.00%)
Worst matchup: Taki (Matches: 3, win rate: 33.33%)

02: Nightmare
Matches: 332
Wins/losses: 277/55
Win rate: 83.43%
Best matchup: Siegfried (Matches: 5, win rate: 100.00%)
Worst matchup: Voldo (Matches: 8, win rate: 25.00%)

03: Mitsurugi
Matches: 347
Wins/losses: 278/69
Win rate: 80.12%
Best matchup: Seong Mi-na (Matches: 5, win rate: 100.00%)
Worst matchup: Nightmare (Matches: 8, win rate: 25.00%)

04: Sophitia
Matches: 371
Wins/losses: 278/93
Win rate: 74.93%
Best matchup: Amy (Matches: 10, win rate: 100.00%)
Worst matchup: Xianghua (Matches: 2, win rate: 0.00%)

05: Hwang
Matches: 340
Wins/losses: 251/89
Win rate: 73.82%
Best matchup: Tira (Matches: 8, win rate: 100.00%)
Worst matchup: Mitsurugi (Matches: 14, win rate: 21.43%)

06: Voldo
Matches: 345
Wins/losses: 230/115
Win rate: 66.67%
Best matchup: Lizardman (Matches: 6, win rate: 100.00%)
Worst matchup: Xianghua (Matches: 5, win rate: 0.00%)

07: Abelia
Matches: 336
Wins/losses: 221/115
Win rate: 65.77%
Best matchup: Amy (Matches: 8, win rate: 100.00%)
Worst matchup: Nightmare (Matches: 13, win rate: 15.38%)

08: Siegfried
Matches: 357
Wins/losses: 231/126
Win rate: 64.71%
Best matchup: Rock (Matches: 5, win rate: 100.00%)
Worst matchup: Nightmare (Matches: 5, win rate: 0.00%)

09: Astaroth
Matches: 359
Wins/losses: 229/130
Win rate: 63.79%
Best matchup: Rock (Matches: 5, win rate: 100.00%)
Worst matchup: Nightmare (Matches: 4, win rate: 0.00%)

10: Cassandra
Matches: 335
Wins/losses: 207/128
Win rate: 61.79%
Best matchup: Seong Mi-na (Matches: 6, win rate: 100.00%)
Worst matchup: Ivy (Matches: 6, win rate: 0.00%)

11: Taki
Matches: 331
Wins/losses: 204/127
Win rate: 61.63%
Best matchup: Maxi (Matches: 9, win rate: 100.00%)
Worst matchup: Sophitia (Matches: 7, win rate: 14.29%)

12: Ivy
Matches: 349
Wins/losses: 214/135
Win rate: 61.32%
Best matchup: Maxi (Matches: 9, win rate: 100.00%)
Worst matchup: Girardot (Matches: 7, win rate: 0.00%)

13: Li Long
Matches: 324
Wins/losses: 197/127
Win rate: 60.80%
Best matchup: Yoshimitsu (Matches: 5, win rate: 100.00%)
Worst matchup: Xianghua (Matches: 8, win rate: 12.50%)

14: Kilik
Matches: 361
Wins/losses: 213/148
Win rate: 59.00%
Best matchup: Rock (Matches: 14, win rate: 100.00%)
Worst matchup: Xianghua (Matches: 11, win rate: 9.09%)

15: Seong Mi-na
Matches: 331
Wins/losses: 189/142
Win rate: 57.10%
Best matchup: Yoshimitsu (Matches: 10, win rate: 100.00%)
Worst matchup: Mitsurugi (Matches: 5, win rate: 0.00%)

16: Zasalamel
Matches: 330
Wins/losses: 185/145
Win rate: 56.06%
Best matchup: Rock (Matches: 5, win rate: 100.00%)
Worst matchup: Xianghua (Matches: 9, win rate: 0.00%)

17: Revenant
Matches: 299
Wins/losses: 165/134
Win rate: 55.18%
Best matchup: Rock (Matches: 10, win rate: 100.00%)
Worst matchup: Mitsurugi (Matches: 5, win rate: 0.00%)

18: Demuth
Matches: 338
Wins/losses: 186/152
Win rate: 55.03%
Best matchup: Rock (Matches: 11, win rate: 100.00%)
Worst matchup: Mitsurugi (Matches: 9, win rate: 0.00%)

19: Raphael
Matches: 331
Wins/losses: 181/150
Win rate: 54.68%
Best matchup: Yoshimitsu (Matches: 6, win rate: 100.00%)
Worst matchup: Maxi (Matches: 6, win rate: 0.00%)

20: Setsuka
Matches: 373
Wins/losses: 197/176
Win rate: 52.82%
Best matchup: Maxi (Matches: 3, win rate: 100.00%)
Worst matchup: Raphael (Matches: 2, win rate: 0.00%)

21: Girardot
Matches: 350
Wins/losses: 183/167
Win rate: 52.29%
Best matchup: Ivy (Matches: 7, win rate: 100.00%)
Worst matchup: Voldo (Matches: 14, win rate: 7.14%)

22: Tira
Matches: 323
Wins/losses: 156/167
Win rate: 48.30%
Best matchup: Rock (Matches: 5, win rate: 100.00%)
Worst matchup: Hwang (Matches: 8, win rate: 0.00%)

23: Maxi
Matches: 352
Wins/losses: 166/186
Win rate: 47.16%
Best matchup: Yoshimitsu (Matches: 9, win rate: 100.00%)
Worst matchup: Taki (Matches: 9, win rate: 0.00%)

24: Cervantes
Matches: 334
Wins/losses: 155/179
Win rate: 46.41%
Best matchup: Rock (Matches: 10, win rate: 100.00%)
Worst matchup: Mitsurugi (Matches: 10, win rate: 0.00%)

25: Abyss
Matches: 362
Wins/losses: 167/195
Win rate: 46.13%
Best matchup: Amy (Matches: 5, win rate: 100.00%)
Worst matchup: Xianghua (Matches: 13, win rate: 0.00%)

26: Luna
Matches: 341
Wins/losses: 152/189
Win rate: 44.57%
Best matchup: Arthur (Matches: 9, win rate: 88.89%)
Worst matchup: Sophitia (Matches: 6, win rate: 0.00%)

27: Arthur
Matches: 340
Wins/losses: 150/190
Win rate: 44.12%
Best matchup: Rock (Matches: 5, win rate: 100.00%)
Worst matchup: Mitsurugi (Matches: 7, win rate: 0.00%)

28: Talim
Matches: 333
Wins/losses: 146/187
Win rate: 43.84%
Best matchup: Voldo (Matches: 3, win rate: 100.00%)
Worst matchup: Raphael (Matches: 1, win rate: 0.00%)

29: Yun-seong
Matches: 370
Wins/losses: 156/214
Win rate: 42.16%
Best matchup: Rock (Matches: 5, win rate: 100.00%)
Worst matchup: Voldo (Matches: 6, win rate: 0.00%)

30: Valeria
Matches: 330
Wins/losses: 132/198
Win rate: 40.00%
Best matchup: Amy (Matches: 12, win rate: 100.00%)
Worst matchup: Xianghua (Matches: 7, win rate: 0.00%)

31: Lynette
Matches: 317
Wins/losses: 124/193
Win rate: 39.12%
Best matchup: Aurelia (Matches: 7, win rate: 100.00%)
Worst matchup: Nightmare (Matches: 7, win rate: 0.00%)

32: Miser
Matches: 347
Wins/losses: 131/216
Win rate: 37.75%
Best matchup: Aurelia (Matches: 6, win rate: 100.00%)
Worst matchup: Mitsurugi (Matches: 9, win rate: 0.00%)

33: Strife
Matches: 342
Wins/losses: 113/229
Win rate: 33.04%
Best matchup: Amy (Matches: 10, win rate: 100.00%)
Worst matchup: Sophitia (Matches: 7, win rate: 0.00%)

34: Yoshimitsu
Matches: 339
Wins/losses: 110/229
Win rate: 32.45%
Best matchup: Amy (Matches: 15, win rate: 93.33%)
Worst matchup: Mitsurugi (Matches: 3, win rate: 0.00%)

35: Lizardman
Matches: 339
Wins/losses: 107/232
Win rate: 31.56%
Best matchup: Hualin (Matches: 9, win rate: 88.89%)
Worst matchup: Mitsurugi (Matches: 14, win rate: 0.00%)

36: Chester
Matches: 336
Wins/losses: 99/237
Win rate: 29.46%
Best matchup: Aurelia (Matches: 8, win rate: 87.50%)
Worst matchup: Xianghua (Matches: 14, win rate: 0.00%)

37: Greed
Matches: 364
Wins/losses: 106/258
Win rate: 29.12%
Best matchup: Amy (Matches: 6, win rate: 100.00%)
Worst matchup: Xianghua (Matches: 10, win rate: 0.00%)

38: Hualin
Matches: 347
Wins/losses: 87/260
Win rate: 25.07%
Best matchup: Strife (Matches: 9, win rate: 77.78%)
Worst matchup: Mitsurugi (Matches: 8, win rate: 0.00%)

39: Rock
Matches: 343
Wins/losses: 68/275
Win rate: 19.83%
Best matchup: Aurelia (Matches: 4, win rate: 100.00%)
Worst matchup: Mitsurugi (Matches: 6, win rate: 0.00%)

40: Amy
Matches: 356
Wins/losses: 36/320
Win rate: 10.11%
Best matchup: Hualin (Matches: 6, win rate: 50.00%)
Worst matchup: Mitsurugi (Matches: 9, win rate: 0.00%)

41: Aurelia
Matches: 307
Wins/losses: 31/276
Win rate: 10.10%
Best matchup: Amy (Matches: 10, win rate: 70.00%)
Worst matchup: Mitsurugi (Matches: 6, win rate: 0.00%)
 
That is extremely fascinating...basing my information on my experience with SC3, Siegfried and Nightmare are usually considered a low tier characters, it's interesting they're both so high up, I personally always felt they were seriously nerfed in comparison to SC2's Nightmare, even though I prefer their new playstyle in SC3, Raphael is another character who was traditionally considered low tier, but relatively high on this list, which surprises me less, because I feel like his SC3 version was a more viable one in comparison to SC2's, Xianghua was an obvious one, she was stupid good in both 2 and 3, another interesting part is the bonus characters, Li Long and Abelia always felt pretty viable, Girardot was a surprise because I thought he would be higher, also, I was sure Greed would be much higher, since he always felt like he's got enough tools to work with, anyways, great work man.
 
Wow, it's just amazing how much time and effort you put into this!

I'm afraid that the AI approach is *scientifically* confounded, though. Reading your opponent, apart from operant conditioning, is mostly based on visual cues which is not what the AI will base its reactions on, but input. This way, attack speed, funky move animation, mix-ups and mind games will not be accounted for. So due to the artificial non-human set-up, external validity of your experiment is heavily flawed. Sorry, dude :(

Nonetheless, this IS rather useful when fighting against AI!
 
You should really change your thread name to "Creating the ultimate AI tier list". I got click baited and im sure others will too.
 
You must log in or register to reply here.
Back