A text about a hot potato, the reasons and above all what we’re doing about it.
Bugs – everyone knows them
Has this happened to you as well? Three seconds before the end of an auction, you place a bid on a worker, outbid all the competitors and BOOM. The money is given back to your account, the second-highest bidding association gets the worker. And none of the developers are interested. Really? Oh, but we are! We just started a new round on the PTR2, which provides us with a lot of additional measurements. Here, we want to identify exactly where the performance weaknesses lie that are causing the lag and these bugs.
Some bugs are caused by a synchronisation issue: your browser client requires some time to send the message, before it goes through the firewall and load balancer to our data centre and is then processed alongside the calculations for 50,000 trains, 1,000 active, clicking players, 50 cities, some worker auctions and much more.
Until a month ago, all of that worked without any serious issues. Then we started to receive reports about second-placed associations winning worker auctions. Whether this is caused by the circumstances mentioned above or your browser is simply delaying the auction countdown – this is no simple matter.
Bugs – how we find the causes
Just as complex as our systems: the Imgur tech stack; source: //imgur.com/blog/2013/06/04/tech-tuesday-our-technology-stack/
This is an overview of the technologies used by the image hosting site Imgur in order to provide the whole world with cat pictures. While we don’t have the billions of visitors that Imgur has, what we do is much more complex than just deliver images. Some of these components are not used by us, whereas other subsystems are more complex. Each of these components has at some point been responsible for errors in the game.
We can’t just grab a magnifier and sit in front of the servers to see where bits are accumulating, and it’s also not enough to just see if memory is filled up somewhere. To identify such issues, we need to make changes to the program, install them on the servers with downtime and then check how the game responds. If we’re not lucky, we may be looking at 10 different measures without finding the actual cause.
Sure, you will find games that have fewer bugs. These are either the blockbusters with a weekly budget similar to Rail Nation’s annual one. Or they are less complex, offer less game depth or less of an interactive world than Rail Nation. For example SimCity BuildIt: the complexity is only within your own city, interaction with other players is limited to trading. In Rail Nation everything is connected to everything else. Your train, your investment and your auction bid make a difference for hundreds of players. That’s the attraction of Rail Nation. Yet for the complexity of the game and our quality assurance, it is a great challenge.
What we’re doing about bugs
We’ve gone to great lengths in recent months to improve the game, both the browser and app version. We have hired developers whose main task is fixing bugs. Soon there will be another bug-fix update for the classic servers. You can follow the developments on the server PTR1. Besides some balancing improvements for the passenger transports, it also includes 59 bug-fixes. Additional bug-fixing is already under way for the USA servers.
We have already noticed a slight improvement in our community’s and tech support’s spirit over the last few weeks. The number of support requests has gone done by more than 50%. We’re not satisfied yet and of course the new version won’t be the answer to all the issues.
The trend of Rail Nation ticket numbers in recent weeks. The amount of tickets has been reduced by 50%.
Rest assured that we do read, see and listen to your feedback and bug reports. When selecting the bugs we are going to fix first and regarding the optimisation of passenger transports, we do listen to you. But we also have to set our priorities. It’s nearly impossible to estimate the time it takes for a bug-fix to be developed. An error may be found in five minutes, but in contrast one can also dig in the code for two days, without finding even the slightest of clues. We won’t halt the development of new content for a week just because an engine is blowing out steam in a square. But everything we do, we do with just one purpose in mind: to make Rail Nation fun for you.
This text was written as a reply to a forum post by WalterL on the German forum. Here you can discuss this text in the forum.
We are looking forward to your feedback!
Kind regards, HarrT
PS: Do you want to help make Rail Nation better?