Recently we had a major problem with the push notifications in the Grepolis app that sadly affected our live version. This was especially bad for our players because it meant the attack alarm wasn’t working, which is a vital part of many players’ tactics. I’d like to give you some background on how this happened, and what we’ve done to prevent it happening again. To start, we have to go back to the original event that triggered the problem.

A security message from Google…

In April we got a message from google informing us that the SSL version used by our app was outdated and contained a vulnerability. We take security very seriously at InnoGames so this became a high priority task and we began to work on it. We started investigating and soon ran into our first hurdle.

Update Adobe Air…

As some of you may know, we use Adobe Air as the base technology for our mobile app. It allows us to write the code once and produce both Android and iOS versions, which is perfect for smaller teams like ours. Updating the core technology of a project can be a risky task for a team, as the smallest change somewhere could affect the behaviour of our app. Usually, we wait several months after an update is released to Air before we update our app, so we can make sure the new version is stable and doesn’t adversely affect our app. In fact for the 2 years I’ve been on the team we’ve only updated the Air version twice before.

But now Google wanted us to update to the very latest version, which had only just been released. We decided that the security issue was important enough to attempt the update, so we did. The code compiled, the app installed, and all our tests passed. Everything looked good, so we released the new version and …

Push notifications stopped!

We started getting reports from players with Android devices that the attack alarm wasn’t working. This is a critical problem so we started investigating straight away. We soon discovered that the new version had changed the way push notifications were configured and so they weren’t working any more.

More updating…

A little background on our notification system, we use the Google Cloud Messaging service to provide push notifications to Android devices. In our app we use a library of code the awesome team at Fresh Planet wrote and shared with the Air developer community. Thankfully, they had provided an update of their code which worked with the new Air version, but this update also changed the way texts, icons, and sounds were customised for the notifications. So we started converting to the new system and after a day and a bit we had the basics working, a Grepolis icon and the text displayed.

Done?

Now we had the choice to keep working to get the custom sounds and lights back, or release the fix as soon as possible. We decided that generic sounds was better than no attack alarm at all so, after testing on as many devices and Android versions as we could find, we released the fix. We still had other important features to work on, so we created a new task to fix the sounds and lights and added it to our schedule to be done in the next release.

Will it happen again?

After the dust had settled, we went back to find out how the problem had slipped through and made it live. We found that we didn’t have any tests that checked if push notifications were working. In fact, most of the devices we use for testing in the office had push notifications turned off because all the beeping and vibrating can get very distracting. But we want to stop this from happening again so we will add checking the push notifications and attack alarm to the list of tests that have to pass before we release a new version.

In summary:

We are sorry that such a major issue made it live, but our new tests should prevent it from happening again, and we hope you can rest easier knowing that the SSL version we use is the most secure possible.