What happened to push notifications?

Recently we had a major problem with the push notifications in the Grepolis app that sadly affected our live version. This was especially bad for our players because it meant the attack alarm wasn’t working, which is a vital part of many players’ tactics. I’d like to give you some background on how this happened, and what we’ve done to prevent it happening again. To start, we have to go back to the original event that triggered the problem.

A security message from Google…

In April we got a message from google informing us that the SSL version used by our app was outdated and contained a vulnerability. We take security very seriously at InnoGames so this became a high priority task and we began to work on it. We started investigating and soon ran into our first hurdle.

Update Adobe Air…

As some of you may know, we use Adobe Air as the base technology for our mobile app. It allows us to write the code once and produce both Android and iOS versions, which is perfect for smaller teams like ours. Updating the core technology of a project can be a risky task for a team, as the smallest change somewhere could affect the behaviour of our app. Usually, we wait several months after an update is released to Air before we update our app, so we can make sure the new version is stable and doesn’t adversely affect our app. In fact for the 2 years I’ve been on the team we’ve only updated the Air version twice before.

But now Google wanted us to update to the very latest version, which had only just been released. We decided that the security issue was important enough to attempt the update, so we did. The code compiled, the app installed, and all our tests passed. Everything looked good, so we released the new version and …

Push notifications stopped!

We started getting reports from players with Android devices that the attack alarm wasn’t working. This is a critical problem so we started investigating straight away. We soon discovered that the new version had changed the way push notifications were configured and so they weren’t working any more.

More updating…

A little background on our notification system, we use the Google Cloud Messaging service to provide push notifications to Android devices. In our app we use a library of code the awesome team at Fresh Planet wrote and shared with the Air developer community. Thankfully, they had provided an update of their code which worked with the new Air version, but this update also changed the way texts, icons, and sounds were customised for the notifications. So we started converting to the new system and after a day and a bit we had the basics working, a Grepolis icon and the text displayed.

Done?

Now we had the choice to keep working to get the custom sounds and lights back, or release the fix as soon as possible. We decided that generic sounds was better than no attack alarm at all so, after testing on as many devices and Android versions as we could find, we released the fix. We still had other important features to work on, so we created a new task to fix the sounds and lights and added it to our schedule to be done in the next release.

Will it happen again?

After the dust had settled, we went back to find out how the problem had slipped through and made it live. We found that we didn’t have any tests that checked if push notifications were working. In fact, most of the devices we use for testing in the office had push notifications turned off because all the beeping and vibrating can get very distracting. But we want to stop this from happening again so we will add checking the push notifications and attack alarm to the list of tests that have to pass before we release a new version.

In summary:

We are sorry that such a major issue made it live, but our new tests should prevent it from happening again, and we hope you can rest easier knowing that the SSL version we use is the most secure possible.

Posted in Technical
11 comments on “What happened to push notifications?
  1. Avatar Mattizin says:

    Hi,
    why does the CMs dont get this Infos?
    They could say to the community and give a brief look of what is the issue and everyone will understand it. This time nobody knew it, it just didnt work and we had 0 information what was going on. This is a really bad communication policy in my opinion, but that seems to be normal for innogames and the grepolis community at the moment :/

    • Nils U. Nils U. says:

      Hi Mattizin,
      i cant tell you how this ended up not being community back at that time.
      Sadly the team internal community managers being present at that time are not in the team anymore to answer that.
      Thats the only thing i can give you, but at least i wanted to not leave that comment unapproved/unanswered on the devbloq.

      Cheers NIls

  2. Avatar daniel says:

    Thanks for your hard work. I am also a programmer and learn a lot from your precious experience.

    • Nils U. Nils U. says:

      Hey Daniel,
      our programmers will be really happy to hear that and i will gladly forward this to them. (The one who wrote the article is currently on vacation^^)
      Thank you and enjoy creating the digital world 🙂
      Greetings

  3. Avatar Stu says:

    Push notifications (attack alarm) some cases are not working for IOS users with App ver 2.117.0, is this issue being worked on

    • Nils U. Nils U. says:

      Hi Stu,
      im checked back with the team it is currently only known for the beta server and we made a fix wish is included in 2.19 Version.
      Maybe you could tell us which gameworld and which country you are playing at. That would help if it is not on beta world.

      Greetings

      • Avatar Stu says:

        Hi Nils, i’m in en90 and based in the UK. Basically as long as the iphone screen isn’t locked the alarm works, soon as the screen is locked i only get the notification and no alarm sound.

        IOS is the latest on the phone and I have checked he push settings and they are all enabled in both the game and the phone. The issue only arose when i updated to 2.117.0

        • Nils U. Nils U. says:

          Hi Stu,
          thanks for your quick response we will invastigate on this.
          Hope we can handle this soon
          Greetings

        • Nils U. Nils U. says:

          Hi Stu,
          our developers checked with different iphone devices on different EN worlds and the always got the notifications+sounds. So we cant reproduce this issue.
          Maybe check the settings of your phone, if you somehow muted, app sounds, this specific map or anything like that.
          If other players you know should have the same issue, tell them to file additional bug reports over the bugtracking system and include as much info as possible. (Type of device, ios/android, World and even player id)
          Greetings

  4. Avatar lynet says:

    now it seems that every time the app i s starting a new sound file is downloaded.

    Before it was possible to change the sound (not evryone like the orginal one..) or use app like notifier pro to change the push sound or just i.e. vibration.

    can this be fixed?

    • Nils U. Nils U. says:

      Hy lynet,
      i talked to one of our mobile devs. As far as the developer i asked told me, our customs sounds have been reimplemented with 2.117.0 Version.
      But custom sounds by users choice have never been in the app.
      If you are refering to something differently feel free to right again.
      Greetings