Civ4 AI Survivor Season 5: Conclusions

This was another season of continued failure for England as none of the three leaders could make it into the playoffs again. Egypt was similarly useless aside from the one Ramesses playoff showing in Season Three and Persia had the ignomity to suffer a double elimination with both leaders showing up - and dying - in the same opening round game. It was an unusually bad year for Russia with Stalin and Catherine unable to make the playoffs and Peter doing precisely nothing in his playoff appearance. Other longtime powerhouses Justinian and Mansa Musa also failed to score a single point while Kublai Khan had his worst showing (and first time not making the Championship game) since Season One. We're down to a mere four leaders at this point who have never scored a single point in all five seasons: Frederick, Hatshepsut, Victoria, and Sitting Bull. Everyone other than Sitting Bull has a realistic chance to grab a win at some point (Hatty and Fred aren't notably worse than Asoka or Washington) but Sitting Bull looks to be uniquely incompetent. Perhaps he can get it done in a high peace weight field someday?

At the other end of the spectrum, we had continued strong performances from several individuals that stood out in Season Four. The two biggest names in that regard were Gilgamesh and Willem, both of whom made the final match in Season Four and returned there again in Season Five. If anyone thought that they had been getting lucky last year, that seems to have been put to rest after both of them scored wins and came close to taking home the overall title. I also have to eat my words from the Season Four Conclusions page as Charlemagne took another pair of second place finishes en route to the Championship. For whatever reason, he's done well enough by now that we can't dismiss Holy Rome as the product of dumb luck. It was an impressive title defense for the King of Burgers. Washington also managed to make a huge leap into the finals despite starting the season with zero points to his name, and although this does seem to have been pretty lucky, it was still a lot better than remaining stuck in the loser's club with Victoria and Sitting Bull. Bismarck similarly got on the scoreboard with a pair of kills and a decent Wildcard game showing. And for familiar fan favorites like Pacal, Suryavarman, Gandhi, Julius Caesar, and Huayna Capac, they were all able to score impressive performances even though they fell short of reaching the championship round.

As always, the kills were heavily concentrated amongst the leaders who made the playoffs. The 29 leaders that fell in the opening round only managed 2 kills between them while the 6 leaders that made the Championship collected 15 trophies. This season was a bit less top-heavy overall with a whole bunch of leaders scoring a pair of kills, 11 different AI leaders ending up with exactly two elimination credits. Going into the finals, we had a three-way tie for the lead in kills shared between Julius Caesear, Gilgamesh, and Alexander (who became the first leader in AI Survivor history to land 4 kills in a single game). That tie was broken as Gilgamesh collected two more victims in the Championship to make him the winner of the Golden Spear award with 6 total kills. This was the second year in a row that Gilgamesh took home the award and another sign of how he's been rapidly charging up the leaderboards during the last two seasons.

The AI ranking list has also been updated to reflect an additional year of competition. As a quick reminder of how the ranking system works, leaders score 5 points for a first place finish, 2 points for a second place finish, and 1 point for each kill. This is a completely arbitrary system, of course, but it seems to do a decent job of recognizing AI performance across different games and different seasons. I included an additional column listing the points scored solely in Season Five to help indicate which leaders were the biggest movers in this season. Unlike in Season Four, where 9 of the top 11 leaders scored at least 4 points, we had a number of the previous top leaders suffering upset losses in the opening round. Only two of our eight Pool 1 leaders were victorious in their opening matches (Huayna Capac and Suryavarman) while half of the Pool 1 leaders failed to advance at all. This meant a number of low scores and an opportunity for some of the trailing leaders to catch up, at least outside of Huayna Capac. The Incan leader managed to establish further separation from the rest of the field and continues to be the top-performing individual in AI Survivor history. For a non-modded game of Civ4 with the default Restricted leaders, Huayna Capac probably is the strongest choice of the group.

The biggest mover from Season Five of the competition was Gilgamesh who continued his charge with another 18 points. All of his finishes and kills have come during the last two seasons after some truly unfair starting positions in the early years of this contest. Gilgamesh had a ridiculous start in Season One (the kind we'd never use now) and then drew two games full of high peace weight opponents in Seasons Two and Three. With more normal games in the following seasons, he's been on a true tear and has to be considered one of the favorites moving forward. Asoka quietly had an excellent season with two first place finishes, and of course Mehmed as the overall champion also had a great showing that quintupled his previous total of 3 points. Further down the list, Washington and Saladin joined the "one good game" club along with past one-off showings from leaders like De Gaulle and Isabella and Lincoln. There are about half a dozen of those leaders at this point and their inability to repeat a single strong performance suggests that they were probably lucky. Repeated and consistent finishes are a better indicator of overall skill as opposed to getting fortunate on a single map.

This was a season of upsets and as a result the points were a bit more evenly distributed when compared to past years. In Season Four, the leaders in the bottom half of the list accounted for 7 points while the leaders in the top half totaled a whopping 128 points. (The same ratio for this season was 106 points against 29 points - still lopsided but not to the same degree.) Big single game performances from the aforementioned Washington and Saladin along with Alexander helped inject some additional points into the bottom parts of the leaderboard. A number of lower-ranking leaders also managed to score a higher than normal number of kills, with Tokugawa and Brennus and Genghis Khan and Bismarck all landing a pair of kills apiece despite not having a top two finish. This helped to avoid the mass concentration of points amongst the top leaders as we'd seen in past seasons.

After five seasons, the average score for an AI leader is now 12.70 points. The leaderboard continues to be top-heavy thanks to the Huaynas and Caesars and Justinians of the world hogging so many points which means that the median score remains noticeably lower at only 9 points. Almost everyone has managed some kind of two top finish by this point in time with only nine leaders still locked entirely out of a playoff appearance. In similar fashion, only six leaders are still without a single kill to their credit: Augustus, Ramesses, Frederick, Hatshepsut, Victoria, and Sitting Bull. All of these leaders share the common features of high peace weight and low aggression scores. They generally like to sit in the corner of the map building wonders and hope to be left alone, with the notable exception of Sitting Bull's bizarre AI personality. They make for a great neighbor in a Single Player game of Civ4 but the evidence is pretty clear that it doesn't work for AI Survivor purposes. Gandhi is the only high peace weight AI who's been consistently able to win using this general strategy.

Another year's worth of data has allowed us to update the Trait rankings for Season Five. This list mostly looks similar to what we had following Season Four, with the same three traits grading out as the best of the group. The biggest difference was Creative moving from just behind Financial and Imperialistic to a little bit in front of both of them. That was due to some excellent seasons from various Creative leaders, particularly Gilgamesh and Willem once again but also Suryavarman and Louis to a lesser extent. This wasn't a great season for Financial with only Huayna Capac and Willem performing well, and it was even worse for Imperialistic as Charlemagne was the only such leader to make the finals. (It wasn't a bad year or anything but Imperialistic was a lot less dominant than we've seen in some past competitions.) Rather than debating the specific merits of these three traits, I think the more accurate conclusion is that the three of them form an upper tier separate from the rest of the group. All of them are obviously useful for AI leaders and can help snowball a game through claiming more land or out-researching the opposition.

There was another clump of six different traits in the middle of the pack that graded out as having roughly similar scores. Spiritual, Protective, Expansive, Aggressive, Industrious, and Organized were all roughly comparable to one another without much differentiation between them. The big shock is the presence of the Protective trait in the middle of that group, with this being an unmodded version of Civ4 where Protective is universally agreed to be the worst trait. Part of this may be due to a bunch of AI leaders having the Protective trait and performing well despite it, thanks to Gilgamesh and Charlemagne and Mao. However, we've also seen that the extra Protective promotions and cheap walls/castles genuinely do seem to help out when the AI leaders inevitably get into stupid wars. At the very least, it's hard to argue that Protective is noticeably worse than the other traits under these cirumstances. Spiritual and Aggressive are essentially situated in the same place as they were previously, average traits that are not especially good or especially bad for AI Survivor purposes. Expansive and Organized are both a little bit weaker for these games, just as they were following Seasons Three and Four, without being enough of a dropoff to cause a huge difference. The AI leaders don't seem to struggle with health or do enough to prioritize workers/granaries to get a lot of value out of Expansive, and their maintenance costs are heavily discounted on Deity, enough to make Organized less useful for them.

The Industrious trait continues to feel overvalued due to the presence of Huayna Capac and Stalin, with all of the other leaders having poor seasons but the trait grading out as average instead of terrible due to the two of them. I think that this is one of the weakest traits for AI Survivor purposes that only looks decent because the best leader in the game (our boy Huayna) happens to have the trait. Stalin came back down to earth this year and if he struggles moving forward the Industrious trait won't keep looking so strong. Charismatic remained in next-to-last status this season, and that was a bit of a surprise because it seems to be helpful for the AIs in terms of managing happiness. Unfortunately none of the leaders with the Charismatic trait did especially well this season, with only Hannibal making the playoffs and then immediately getting plastered by Gilgamesh. This trait needs Cyrus to go on another run to boost its ranking as it's increasingly looking like one of the weakest options for AI Survivor purposes. There was a big gap between the glut of traits in the middle of the pack and then the Charismatic/Philosophical duo at the bottom.

This year the Philosophical trait once again graded out as the worst in the competition for the second year in a row. It's been a precipitous fall from the Season One version of these rankings which had Philosophical as the best trait overall. That was under the old Finish Ranking system that heavily valued silly stuff like not getting eliminated in the Wildcard game so it's not a direct comparison, but it's still noteworthy how the trait has collapsed in value over the following seasons. Philosophical is a poor trait for the AI leaders because they inevitably run so many specialists that they always get lots of Great People regardless. Plus the AI has no idea how to leverage individual Great People for timed Golden Ages, and of course we turn off Tech Trading to remove the value of lightbulbing followed by tech brokering. The Philosophical trait also happens to be paired with lots of pacifistic leaders with high peace weights, and as we've seen repeatedly, those are some of the worst-performing AI personalities under our settings. Philosophical is a solid trait overall in Civ4 but it clearly struggles under the games that we run.

We also continue with our tradition of looking at the average score of the leaders making up every past match (even as this list is starting to get absurdly large). This allows us to see which games had particularly strong or weak fields, as well as revise our opinions about some matchups from past seasons that might have looked different at the time. Keep in mind that the average leader score after five seasons is about 12-13 points, and therefore games above that have stronger fields and those below that have weaker fields. (For the statistically inclined, the standard deviation of the opening round matches is roughly 2.68.) The weakest opening round field in AI Survivor history is now Season Three Game Eight, De Gaulle's one and only victory that featured double zero-point leaders Hatshepsut and Frederick to drag down its rating. The score of 8.43 is almost two standard deviations below the median, something that we would expect to appear only about 5% of the time in a purely random setup. That game felt like a poor group of competitors at the time and the ranking system backs up that impression. Other weak opening round games include Boudica's win in Season One Game Three (8.50) and a Gandhi snooze-fest in Season Three Game Two (8.71). The worst game of all time under the current ratings is Season Two's Wildcard game (8.18) thanks to having so many high peace weight builders present. We're unlikely to get another game with 11 leaders any time soon.

As far as the best collection of opening game talent, we have three games that grade out with a rating above 17.00. The highest score comes from Season Three Game Seven at 17.83, the infamous Wang Kon troll job that somehow rates as the best opening round game ever. The score might seem hard to believe but the game had three different champions (Huayna, Charlemagne, Mehmed) along with middling performers Wang Kon and Tokugawa. This a result that "feels" wrong even if the math is clear. The other top games were Season One Game Seven (17.50) where the double Russian leaders crushed everyone else and Season One Game Two (17.00) which was the first-ever "group of death" match. We're much less likely to get a lopsided opening round these days since we implemented the seeding system and therefore it's no surprise that some of the more unbalanced opening round games came from early in the competition's history.

With regards to the playoffs, the perception was that Playoff One was an unusually weak group and the numbers back that up. Season Five Playoff One rates as the worst playoff game of all time with a score of only 13.50 points. There were three different opening round games this year alone that graded out with a better field! The leaders in that game were lucky to have such an easy path to the finals. The other two playoff games were much more typical, with slightly higher ratings than the five year playoff average. This upset-heavy season also delivered the lowest-scoring Championship game ever as well, although not by much in comparison to the fairly weak Season One Championship game. The second and third playoff games both felt like they had better fields than the finals and the numbers confirmed that this was indeed the case. By way of contrast, last year's Season Four Championship looks significanly stronger in retrospect with another fine season from Gilgamesh, Willem, and Charlemagne added to the data. And at the extreme upper end of the spectrum, the Season Three Championship increasingly looks like a legendary collection of AI Survivor talent with an average score of 31.33 points! That game had five of our top seven leaders in the total ranking along with laggard Pacal who "only" grades in at 12th place out of 52 leaders. I still think that most of those leaders would have mopped the floor with the Season Five Championship entrants.

Mehmed of the Ottomans was the surprising winner left standing at the conclusion of a thrilling final game in the Season Five Championship. Unlike every previous winner of the competition, he had never taken first in a single game before he edged out Gilgamesh's spaceship in the finals by a nose. However, there was some evidence to suggest that Mehmed was a decent militaristic leader due to the way that he had come up short in several previous performances. Mehmed scored two kills in his Season One outing when he was drawn into the infamous "group of death" and played a strong game for the first 300 turns before getting eliminated by an out-of-control Mansa Musa. He fell afoul of Mansa again in Season Two, and then appeared in the "highest rated opening round match ever" in Season Three where he scored another early kill and looked to be on the verge of snowballing before collapsing under attack from a Charlemagne and Wang Kon tagteam. In Season Four Mehmed flopped on his face despite what was seen as an excellent starting position, but Wyatan's closer look at this map in the alternate histories data suggested that it wasn't actually a great capital at all. There was no early copper resource to be had and Mehmed was almost always out-performed by the (much better than we thought at the time) nearby Willem. Similarly, in Season Five Mehmed picked up two more kills in games where he didn't win, and another alternate histories investigation of Playoff Game Three revealed that he had suffered from an unusually weak outing in the game that we watched on Livestream. Mehmed had actually been the second-strongest AI on that map and had nearly always been in contention for victory, not the pathetic sad sack tagalong figure that we experienced in the real match. Mehmed had the most kills of any leader still lacking a victory and that suggests that we were undervaluing him a bit. He's definitely not one of the best leaders - the inability to declare war at "Pleased" is a real problem - but Mehmed is at least decent as an AI leader. He probably falls somewhere in the 20-30th place range and had the good fortune to see everything come together at the right moment. At the very least, he didn't just avoid losing the way that Charlemagne did at the end of Season Four.

Anyway, I'd like to give thanks here to everyone here who took part in Season Five of AI Survivor. Even if it was nothing more than spending two minutes to submit a prediction, the competition wouldn't have been the same without all of your contributions. We had our best turnout yet this season, with about 225 predictions on average for each game and more than 200 viewers (!!!) watching along on the Livestream. We peaked at 258 live viewers during the Championship:

This was by far the most viewers that I've ever had watching outside of my IPL casting days for League of Legends. Viewership and prediction contest entries were both up about 50% as compared to Season Four and I continue to be amazed at how many people are interested in watching the computer play against itself. This event simply would not be what it is without the lively banter in the chat taking place alongside the action. I am surprised again and again at the brilliant insights that the viewers come up with in their predictions, along with the hilarious and inane running commentary that makes it such a pleasure to put these games together. I think that the format we used this year was a huge success and the removal of the Apostolic Palace felt like a big improvement. Over the upcoming months we'll take some time to look at alternate histories for other games from Season Five and think about any additional fixes that we might add for the future. With any luck, we'll be back again in 2021 for Season Six - thanks again everyone!