Last Saturday we had the first ever Lean Poker Tournament, an event much like a code retreat, but with a slightly different format and purpose. A lean poker tournament's aim is for participants to practice concepts related to lean start ups and continuous deployment. A poker team is a small group of developers (ideally 4 people forming 2 pairs) whose aim is to incrementally build a highly heuristic algorithm within a one day time frame that is just smart enough to beat the other robots. Professional poker robots are developed for years, so the purpose is definitely not to come up with something really smart, but to be the smartest among the current competitors. With this in mind teams can come up with simple to implement heuristics, examine their effect on game play during training rounds, and then improve their algorithm in a similar lean fashion.
Since it's been the first such event, the format is still quite flexible. My current vision is that eventually robots will play consecutive rounds of sit'n'go tournaments almost continuously. The robots will be automatically redeployed from a git repository master branch by the framework before each sit'n'go. The winner of each sit'n'go gets 5 points, the second gets 3 points. If a player does not respond for a round it gets punished by loosing a point. This version lets teams do many experiments, and also forces them to be careful when pushing code to master.
The teams have 60 minute sessions while they are allowed to code. After each session there is a break, during which the last game that was played by the robots is shown on a projector. This is the part, where the teams can root for their robots. After the break there is a quick retrospective stand up meeting that looks back on the previous session.
The first event had a bit different format. Since I wasn't sure if lean poker was to be successful I only aimed at a minimum viable product, and I settled for a manually running the server from time to time. This turned out to be a good idea, since the most important lessons were related to deployment of the robots and communication between them.
However I'm really glad that I did not compromise on the web application that allowed us to view games like one would view them in an online poker room. I did compromise there on code quality quite a bit, but since it's a small piece of code, and I had like 5 hours to do it, I think it was a good call. The moments spent in front of the projector were the most exciting and most memorable. Based on the feedback I got from participants this is probably one of the most important parts of the event.
The first and most painful lesson was that Thrift was a really-really bad idea. When I started the project I was impressed by the concept of an RPC framework like Thrift, and based the entire project on it. As the project progressed first I started to have problems with the insufficient documentation and the inconsistent design of Thrift. I won't go into details here (maybe in another blog entry) but it turns out that using Thrift is harder than implementing a RESTful service. However that alone would not have made me decide that I would replace Thrift with REST before the next event. The main problem was that the inconsistencies, and insufficient error messages resulted in most of the morning getting wasted on trying to get Thrift to work.
The second lesson was that the players have to run on the same machine as the the server. On events like this the host usually only provides a Wifi network. Since the robots did a lot of synchronous communication the large ping times associated with Wifi slowed the game down to a point where the players were essentially responding at the same speed a human player would have. The first round we conducted was during lunch and took about half an hour.
For the second round I cloned the repository of each team, and the sit'n'go's could run in seconds. It's sort of lucky that I didn't automate the starting of sit'n'go tournaments yet, since if I did that, then it would have been a bit harder to make that change.
Another interesting lesson was design related. With Thrift under my belt it seemed natural to send events to the players. This mistake proved to be significant in different ways. The most obvious problem was that this resulted in a lot larger communication footprint for the application. Every event had to be broadcast synchronously to each player and that also contributed to the slow response times.
The second problem with this approach was that the players had to maintain state for themselves. In case of the Java and Python teams this was only a minor inconvenience, but the PHP team ran into one of the Thrift gotchas: while Thrift servers on other languages act as singletons, the PHP server is stateless, and every remote function call is performed on a new instance of the handler. Actually it's even worse than that: each request is handled by a new process and not even global state is preserved, so the PHP team had to persist state on disk.
This design failure can be summed up like this: I've been thinking of players as components and not as services. While components are organized around boundaries of change, services should be organized around resources. While components should communicate through small messages, for services it is better to communicate through documents. This is a more general lesson that one should keep in mind, and one that I had to learn the hard way.
If you wish you had come...
The good news is that the first event was a success, so I can't wait to organize the next one. However I did learn some important lessons, and I'd like to modify the croupier app. to reflect on those. I hope I can get it done by March, and if the next event goes well, I plan to do a small road trip around Europe, and take it to other cities. This would help me finalize the format while also promote it among software developers.
So if you'd like to participate at a Lean Poker Tournament, feel free to come to one of the next events in Budapest, or help me organize one in your city. I also appreciate any help with developing the poker croupier application.