Written by Guy Haworth and Nelson Hernandez
Reading, UK and Maryland, USA

Corresponding author: [email protected]

This is the latest in our series of analytical articles on past Tcec events. The main text can be read below on this webpage, and at the bottom you will find a link to the full layouted article in pdf format, including the important tables, graphs and images.

TCEC is very grateful to the authors for their kind permission to publish these substantial and scholarly analyses of its events!

Introduction

TCEC Season 14 started on November 12th 2018 and introduced a number of changes from TCEC 13 (Haworth and Hernandez, 2019b). An enlarged Division 4 featured twelve engines and seven newcomers to accommodate the increasing interest in computer chess and this competition in particular. The other divisions remained eight strong. The five divisions played two or more double round-robins (‘DRR’) each with promotions and relegations following. Tempi gradually lengthened from ‘Rapid’ to ‘Classical’, and the Premier division’s top two engines played a 100-game match to determine the Grand Champion.

The trio of STOCKFISH, Komodo and HOUDINI have dominated the TCEC medals for several seasons and a key point of interest was whether others would reach the podium. Leela CHESS ZERO and ETHEREAL were certainly expected to perform well in Division P, having shown remarkable improvement in the previous few months. KOMODO MCTS was a dark horse.

There were a few nudges to TCEC’s adjudication rules. Draw adjudication could be invoked after move 35 (rather than move 40) and the two engines had to both evaluate within ±0.08 (rather than ±0.05) for eight consecutive plies and with plycount≠0. While draw-adjudication requirements were relaxed, win-adjudication requirements were tightened. Engine evaluations had to be outside ±10 (rather than ±6.5) for ten consecutive plies (rather than eight); plycount was not a factor. This change was welcomed by those of us who wanted to see a clearer demonstration of superiority on the board: it will be interesting to see how long it prolongs the decisive games and what mysteries remain.

The common platform for TCEC14 consisted of two computers. One was the established, formidable 44-core server of TCEC11-13 (Intel, 2017) with 64GB of DDR4 ECC RAM and a Crucial CT250M500 240 GB SSD for the EGTs. The ‘GPU server’, a Quad Core i5 2600k, was sporting Nvidia (2019) GeForce RTX 2080 Ti and 2080 GPUs for those engines which could exploit them.

The engines

Season 13 competitors BOBCAT, DEUS X, HANNIBAL and SENPAI rested for this TCEC season. TCEC welcomed first appearances for engines DEMOLITO, KOMODO MCTS, PIRARUCU, ROFCHADE, SCHOONER, SCORPIONN and WINTER, see Fig. 1 and Table 1.

Division 4: 2 DRRs, 4 round robins, 264 games, 30′+10″/m

As for TCEC12/13, each engine played both White and Black from four-ply openings defined by the second author here. The results are as in Table 2: ‘P%’ is the %-score and ‘ELO±’ is the change to the engine’s nominal ELO based on its performance. Generic stats are in Tables 9 and 10.

Online interest naturally focused on the newcomers, especially KOMODO MCTS (Chessdom, 2019), a further innovation from the Lefler/Kaufman camp. The engines had a wide range of ability leading to only 34.1% of games being drawn: those given a default ‘TCEC-entry ELO’ of 2900 ranged across the field. WINTER was always headed for a demotion spot. SCORPIONN clearly was not ready for the contest and even though it disconnected eight times, it did not impact the ranking elsewhere. The bottom three missed TCEC Cup 2 (Haworth and Hernandez, 2019c). The three engines promoted were clearly ahead: KOMODO MCTS, ROFCHADE and NEMORINO.

Division 3: two DRR phases, 14 rounds, 112 games, tempo 30′+10″/m

Again, the eight engines involved played both sides of 14 prescribed four-ply openings. With GPU operating temperatures more stable, LCZERO was expected to do well after its performance in TCEC Cup 1 (Haworth and Hernandez, 2019a) and it did not disappoint, see Table 3. KOMODO MCTS also distanced the rest of the field and continued on up the divisions. Crashes remained a problem: this time, HANNIBAL incurred five. In game 26/7.2, ROFCHADE disconnected in a 7-man tablebase drawn position. In g93, NIRVANA retained the KBNPKRN draw for 101 moves but claimed the 50-move draw with 165. Bd6 – which loses to 165. … Nf7+ 166. K~ Nxd6. Do chess programs do irony?

Division 2: two DRR phases, 14 rounds, 112 games, tempo 30′+10″/m

Game 64/16.4, KOMODO MCTS – LEELA, ended in a rare stalemate on m172. Game 93/24.1, NIRVANA–LEELA, was drawn at position 115b but a mate for Black in 29 moves when the 50-move draw rule intervened. Demoted GULL beat BOOOT and BOOOT beat LEELA which otherwise moved smoothly away to win the division again, see Table 4. The silver medal went to KOMODO MCTS, courtesy of one less loss to LEELA than XIPHOS and one more win to the rest of the field.

Division 1: two DRR phases, 28 rounds, 112 games, tempo 60′+10″/m

The penultimate game 28.3/111 was the longest ever for TCEC Division 1 at 308 moves: ‘new wave’ LEELA versus ‘old guard, oldest brand’ FRITZ 16. The win is routine enough with rook and passed pawn against a half-sighted bishop but endgame solver FINALGEN (Romero, 2012) sees 20 moves before a clear win, a line that results in mate on move 337 at best (Haworth and Hernandez, 2019d).

GINKGO surprisingly crashed four times and was disqualified so the formal results are slightly different from those of Table 5 even if promotions/relegations are otherwise unaffected. FRITZ never saw a win in this company and also was demoted to division 2.

Division P, three DRR phases, 42 rounds, 168 games, tempo 90′+10″/m

The line-up for Division P had only a semi-familiar look. After the TCEC13 podium trio of STOCKFISH, KOMODO and HOUDINI, we had the other survivors FIRE, ETHEREAL and ANDSCACS. Interest however centered on the newcomers LEELA CHESS ZERO and KOMODO MCTS, both bringing MCTS search to the game. The contest was three DRRs rather than the four of TCEC13.

After the first round-robin, STOCKFISH had jumped out into the lead with four wins. After the first DRR, with colour-bias eliminated, STOCKFISH maintained a healthy lead and remained unbeaten, a feat shared with KOMODO and LEELA. Was the TCEC podium about to change? KOMODO MCTS had disconnected and lost twice against KOMODO in drawn positions. A third disconnection would be bad for both engines: disqualification for MCTS and elimination of Komodo’s crash-wins from the table.

Game 64 saw STOCKFISH beat KOMODO, opening the door for LEELA. In game 68 at the foot of the table, ANDSCACS beat ETHEREAL with Black. At the half-way point, LEELA was edging the contest for second place and remained unbeaten. The fourth round-robin saw LEELA consolidate its second place with four straight wins against the tail including one as Black against ETHEREAL. The competition for second place remained open as STOCKFISH finally ended LEELA’s unbeaten run in the last RR4 game, g28.4/112.

The fifth round-robin saw plenty of drama. LEELA lost as Black to both KOMODO and FIRE, the first having serious tie-break significance and the second being seriously unexpected. GPU fan-settings were thought to be a contributory factor but not enough to trigger replays. In game 33.1/129 v HOUDINI, KOMODO MCTS disconnected for a third time, was disqualified and relegated with its games discounted. Hopefully, Mark Lefler will sort out the technical problems for TCEC15. This restored LEELA to second place. With one round-robin to go, adjusted scores at the top were STOCKFISH well clear on 21, LEELA 16.5, KOMODO and HOUDINI 16. The second relegation spot was between ETHEREAL on 11.5 and ANDSCACS on 11.

Every win was now going to be a major event, especially as the last round of 28 games started with seven draws. KOMODO as White lost to STOCKFISH in g37.4/148. Both LEELA and KOMODO beat FIRE. In the penultimate game, KOMODO beat ANDSCACS: ETHEREAL breathed again, having narrowly survived without a single win in this division. In the last game, a cliffhanger, STOCKFISH searched the endgame tables a thousand times more than LEELA and thought it had a feasible advantage, but LEELA held out in KRPPKRP to draw on move 93.

The raw figures of Tables 6 and 7 need adjustment because KOMODO MCTS’ disqualification flipped the ranking at both ends of the table. In fact, STOCKFISH ultimately had 25 points, LEELA 20, KOMODO 19.5, ETHEREAL 14 and ANDSCACS 13.5. The ‘big three’ became the ‘big four’ but the Shannon-AB engine mould was cracked again: the still-improving LEELA had remarkably progressed from Division 3 all the way to the TCEC Superfinal.

As in TCEC13, a knockout event was interposed between this tournament and the Superfinal. Would the LEELA team roll out an improved network in preparation for the big finish? A hint came in a ‘bonus match’ between a more recent ‘LEELA 32585’ and ‘STOCKFISH 8’, the latter having only 12 threads and a 4M hash-table. This was an echo and ‘simulation’ of the ALPHAZERO–STOCKFISH match: LEELA won +24=71-5. We reported on TCEC Cup 2 separately (Haworth and Hernandez, 2019c).

The TCEC14 Superfinal match: 100 games, tempo 120′+15″/m

TCEC’s ELOs suggested a STOCKFISH win by eleven. However, both engines came to the board in new versions: the match was now STOCKFISH v190203 versus LEELA v20.2-32930. There was bound to be a clash of styles occasioned by the different modes of evaluation and use of hardware. This dynamic was eagerly anticipated with viewer numbers often topping 2000. Jeroen Noomen (2019) again created a suitable opening book, aiming as before for at least 20% decisive results. Assaf Wool returned from his ‘TCEC Cup break’ to comment on all the games. GMThechesspuzzler and Kingscrusher were active on Youtube (Wool (2019) picked out positions from games 7, 8, 10-11, 13, 16-17, 20-22, 25, 27, 29, 35, 41, 49, 53, 55, 58, 63, 65-66, 71, 75, 80, 85 and 87. Kingscrusher (2019) commentated on games 7, 10-11, 13, 16, 17, 53, 66 and 85. Games 2, 7-8, 13, 17 20, 29, 49, 65-66, 80, 85 and 100 were covered by GM Thechesspuzzler, 2019). Soren Riis provided the authors with detailed analysis of games 7-8, 20-22, 65-66 and 71 which we provide via our pgn file for reader convenience rather than here. GM Matthew Sadler (2019), having analysed the STOCKFISH–ALPHAZERO games (Sadler and Regan, 2019) has also contributed his own view of this Superfinal.

The play and the results did not disappoint. STOCKFISH opened its account with wins from games 7 and 10 but LEELA replied with wins from games 11 and 13. There were twelve wins in the first thirty games, a hit rate of exactly 40%, see Table 8 and Fig. 2. At this point, the score was 15-15, suggesting that this would be the closest TCEC Superfinal since Season 5 in 2013 even though LEELA had never led. The same situation appertained at 24-24 after a run of 19 draws (not a record: the TCEC8 KOMODO 9.3x – STOCKFISH 021115 Superfinal games 14-37 and 47-71 were all draws). At this point, LEELA dramatically jumped out front with wins in games g49 and g53. This lead held until game 80 which STOCKFISH won. Ultimately, it was the single 0-1 win in another sea of 19 draws that allowed STOCKFISH to retain the title. Each game was closely contested with average length being one ply short of 100 moves – and not just because LEELA was reluctant to visit the draw zone.

Of course, suitably equipped grandmasters could write a book about this entirely gripping match and this would be most welcome. Here, we can only pick out a few chessic highlights which perhaps complement the analyses of the commentators above.

The hints from the evaluations of STOCKFISH suggest that it welcomed LEELA’s 15. Bb2 (g07), 51. … Be3 (g08, a missed win), 34. Kf1?? (g21) and 31. … Qd6 (g22). In game 35, 29. Ke1 rather than h7 seemed to lose LEELA’s winning advantage. Game 58 was adjudicated with a rare ‘mate in one’ on the board: the camera cut away just before the blow was struck. Game 63: LEELA was happy to trade pawns for position as early as eleven moves into the play. STOCKFISH did not see a serious problem until six moves later. LEELA create a passed pawn despite being three pawns down and this led to a crushing 41-move win, the shortest of the match.

If there was a pivotal juncture in this Superfinal, it was games 65-66 – a crucial one or two-point swing to STOCKFISH. In game 65, LEELA missed a KNP(c4)P(d5)KBP(c5) win with the winning capture admittedly 26 moves down the line (de Man, 2018). STOCKFISH clearly saw it was lost and LEELA would have been awarded the win under the TCEC13 ‘6.5+’ win-adjudication rule. LEELA was within 11 ply of winning with 9 ply to go and it is worth speculating as to how soon it would have found the winning idea, K on b5/c6 before Nxc5, had the plycount not intervened. Game 66 had to be restarted after two server crashes before LEELA – lost. Had it been possible to return to the game-state after the last completed move, the temperature of the partisanship in the chat room would have been lower. A minor cost, but transaction-checkpoint/restart might be applicable here.

Game 85 was the final win: the 12-move King’s Indian opening had already defined the major asymmetry of Queen versus BBPP. LEELA went from apparent equality to negative territory by move 25. Ultimately, LEELA’s QR were unable to prevent mate by a BBNNPP team, only five moves away when the referee stepped in. Game 86 was the longest ever TCEC game at 362 moves.

The Bonus 4-way and 2-way Rapid events

TCEC treated us to two bonus events at the Rapid tempo of 12′+3″/move. The first featured the top four - HOUDINI, KOMODO, LEELA CHESS ZERO and STOCKFISH: 20 DRRs, 40 round robins, 120 rounds and 240 games. STOCKFISH had a good first half and was never headed even if pursued closely by LEELA. HOUDINI and KOMODO tailed off, eventually in that order as KOMODO fared poorly in the second half. ELO-predicted net scores were +9/+1/-2/-8 but ‘actuals’ were +12/+6/-7/-11. The longest wins were g116.1 (1-0, 139 moves) and g37.2 (0-1, 125m): the longest draw, g12.1 (318m). Game 18.1 between Leela and Komodo was something of an anti-climax as a 3x-repetition draw after ten played moves. Full details are included with the repository e-version of this note (Haworth and Hernandez, 2019d).

The second event was a 100-game STOCKFISH–LEELA match from the initial position: no prescribed openings. LEELA won 16-4, perhaps by being single-minded about its openings (Wool, 2019).

Summary

The Google DeepMind company in St. Pancras, London have been remarkably open in sharing the core ideas of their intelligence initiative. In the year it has taken for DeepMind’s papers on ALPHAZERO (Silver et al, 2017/18) to mature and satisfy the referees, we have seen TCEC invest in Nvidia GPUs and foster several innovations going beyond the classic Shannon (1950) minimaxing AB model of a chess engine. We have seen a leading chess-engine author, Mark Lefler, move his focus successfully from top engine KOMODO to KOMODO MCTS (Chessdom, 2018). With one less technical break, this engine would have come all the way through the divisions to fully justify its place in Division P at the first attempt.

We have also seen a community come together to support and train the open-source LEELA CHESS ZERO echo of ALPHAZERO. Again, this has been rewarded by success, and how. LEELA edged out KOMODO and HOUDINI to take the challenger’s place in the Superfinal here. It was not expected to beat STOCKFISH but came within one game of drawing the classic phase.

Chess24 and Chessbomb, with its useful colour-coding of moves, covered the TCEC14 Superfinal so we were treated to kibitzing by three different, objective but hardly neutral versions of STOCKFISH. The Twitch TCEC channel claims that viewers’ computers have to date had a window open to TCEC Seasons 10-14 for a total of over half a million hours.

REFERENCES

• CPW (2019). https://tinyurl.com/icga046. The Chess Programming Wiki website, including biographies of engines, authors and developers.
• Chessdom (2018). http://tinyurl.com/icgak034. Interview with Mark Lefler and Larry Kaufman.
• de Man, R. (2018). http://tablebase.sesse.net/syzygy/. Site providing sub-8-man DTZ50″ EGTs.
• ‘GM Thechesspuzzler’ (2019). https://tinyurl.com/icga056. Superfinal video-commentaries.
• Haworth, G. McC. and Hernandez, N. (2019a). http://centaur.reading.ac.uk/80284/. TCEC Cup 1. This note plus annotated statistics and pgn files. Submitted to the ICGA Journal.
• Haworth, G. McC. and Hernandez, N. (2019b). http://centaur.reading.ac.uk/78820/. TCEC13: the 13th Top Chess Engine Championship. Submitted to the ICGA Journal.
• Haworth, G. McC. and Hernandez, N. (2019c). http://centaur.reading.ac.uk/81390/. TCEC Cup 2. This note plus annotated statistics and pgn files. Submitted to the ICGA Journal.
• Haworth, G. McC. and Hernandez, N. (2019d). http://centaur.reading.ac.uk/82052/. TCEC14: the 14th Top Chess Engine Championship. Submitted to the ICGA Journal.
• Intel (2017) https://tinyurl.com/icga042. Intel’s specification of the XEON® E5-2699V4 processor.
• Kingscrusher (2019). http://tinyurl.com/icgaj057. Superfinal video commentaries.
• Noomen, J. (2019). https://tinyurl.com/icgaj054. JN’s approach to the Superfinal openings.
• Nvidia (2019). https://www.nvidia.com/en-gb/geforce/graphics-cards/rtx-2080-ti/ GEFORCE GTX 2080 TI GPU specification and benchmark performance data.
• Romero, P. P. (2012) https://tinyurl.com/icga013. FINALGEN: tutorial, download and forum.
• Sadler, M. (2019). The TCEC14 Computer Chess Superfinal: a perspective. Submitted to the ICGA Journal and https://tinyurl.com/icga055.
• Sadler, M. and Regan, N. (2019). Game Changer: AlphaZero’s Groundbreaking Chess Strategies and the Promise of AI. New in Chess. ISBN 978-90-5691-818-7.
• Shannon, C. E. (1950). Programming a Computer for Playing Chess. The London, Edinburgh and Dublin Philosophical Magazine, 41(314), 256-275. https://doi.org/10.1080/14786445008521796.
• Silver, D. et al (2017). Mastering Chess and Shogi by Self-Play with a General Reinforcement Learn-ing Algorithm. arXiv: 1712.01815.
• Silver, D. et al (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140-4. doi: https://doi.org/10.1126/science.aar6404.
• Wool, A. (2019) http://mytcecexperience.blogspot.co.uk/ TCEC blog.

Full article

To read the full article in pdf, click HERE

published March 11, 2019