“Borderline bulletproof” security for API keys

In episode 5 of my Keep in Touch podcast I shared some thoughts on how I currently think about protecting credentials, API keys, and secrets (summarised as keys for the rest of the post) for 1st and 3rd party services and APIs. This blog post is a recap of those ideas and also a call for feedback and input from you.

The risks

The problem I am talking about addressing, is related to the fact that many applications embed keys in their binaries. These keys are required to authenticate (and authorise) the app, the user, or both with remote services. These services could be public or private APIs. Similar considerations apply if these APIs are 1st party or 3rd party.

If these keys are bundled with the app, a nefarious attacker could extract them and then impersonate the developer.

It’s not my intention to teach bad actors how to execute their plans, but I do want to give a few examples of some attack vectors that could lead to sensitive information falling in the wrong hands:

  • the nightmare scenario: the attacker could run a man-in-the-middle attack by running a WiFi hotspot or could be operating one of the nodes on the internet between the device and the APIs the app connects to
  • the easier than you’d think scenario: the attacker could run an on-device proxy (such as the brilliant Charles for iOS app that I use daily for non-nefarious reasons), and monitor all traffic
  • the attacker could grab the app binary (ie. apk or ipa) and do a string search for key-like strings
    Any of the above could lead to the attacker obtaining the keys that the app uses to authenticate itself against remote servers. Once they get these keys they can start impersonating the app and its users.

The hypothesis

I believe that by not including any keys in the app’s binary and by provisioning these credentials only via a channel owned by the operating system that uses SSL pinning, the risks mentioned above can be reduced or even avoided.

SSL pinning is essential in order to prevent an attacker from inspecting the traffic between the client app and the remote endpoints.

Animated GIF showing the way keys can be securely transmitted from cloud to app to server. This flow is described in detail below

This animation attempts to illustrate the approach I am suggesting:

  1. Provision the app without any keys
  2. Attempt to detect if the app is running on a jailbroken / rooted device, and bail if that is the case
  3. Upload the necessary keys to the Trusted Cloud provided by the Host operating system (i.e. CloudKit for iOS apps) using another app (i.e. console, browser, etc)
  4. Enforce read-only access to these keys for the client app
  5. Upon (first) launch, fetch the keys from the Trusted Cloud
  6. If the keys must be stored, only store them in a keychain or in a secure enclave
  7. Use the keys to communicate with the Server using an SSL pinned connection
  8. Should the keys become compromised, they should be replaced, and the client apps should naturally fetch the new keys. For this reason, getting the new keys should not require the previous keys.

The steps above rely on the fact that the mobile OS provides a cloud service that uses SSL pinning in order to communicate to the client apps. Without it, there cannot be a secure key provisioning mechanism and the entire flow falls apart.

The checklist

Often times the security considerations come in “late” or “after the fact”. To assess the state of affairs, and to add more protection going forward, I would go over the checklist below:

  • Limit the number of people who have access to the credentials. Replace the credentials when these people leave the job.
  • Never embed the keys in the codebase
  • Never grant the build (or distribution) servers access to the keys
  • Only trust the cloud service that belongs to the platform vendor for your app (i.e. CloudKit for iOS apps, Firebase for Android apps.)
  • Avoid storing the keys anywhere (on the device) unless really necessary
  • If the keys must be stored, then the only acceptable place is the Keychain
  • Avoid bundling black-box libraries. If the libraries use networking, inspect all networking traffic
  • Do not use a 3rd party networking library unless absolutely necessary. Always check the source code of such libraries
  • Never be the largest consumer of any one technology, framework, or library
  • Only load the keys at the code level that really needs them. Do not pass them through the business layer, particularly if there are 3rd party analytics or logging libraries bundled with the app
  • Obfuscate the code and avoid obvious naming strategies for the keys
  • Check the app’s signature / hash or anything else similar that guarantees that the binary has not been tampered with
  • Be mindful of jailbroken or rooted devices but never change the code in order to support them
  • Do a quid-pro-quo vulnerability analysis with a trusted partner (make sure dinner or beer is something at stake if an issue is found)

If you have more to add to this list, I would love to hear from you!

Closing comments

I think of myself as a security minded person, but I am by no means an expert. I offer no guarantees that implementing the steps above is going to remove or prevent any security related problems for a system I have not seen before, but, based on my current knowledge, I do believe it is the closest thing to a “borderline bulletproof” approach to securing keys and credentials that native apps use.

Lastly, I sincerely hope that, if you read this and have identified a flaw in my reasoning, or if you can spot an oversight, you will share with me (ideally with a solution in tow) so I can update the article.

The free and easy way to start a Podcast

Setting up a Podcast in 2019 can be fast, free, and painless. More importantly, the distribution and maintenance of the show can be (almost) fully automated. This post covers the steps I took to launch KeepInTouch.fm together with my co-host, Todd.

MVP – Minimum Viable Podcast

  • First we decided what our show is about. Keep in Touch covers topics from two main areas: Apple/Google tech platforms, and raising kids
  • To solve the hosting and publishing of the show we set up an account on Anchor.fm
  • Lastly, we recorded and published the show

That’s could’ve been the proverbial it, but since both Todd and I are tech people, we were not satisfied, yet. Why not? Because of the details…

Past the basics

We wanted the show to have a home that we could customise, personalise. We also wanted to be able to understand and interact with our audience. To achieve this we made did a few more things:

  • We put a little effort in the show artwork, we created chapters for the shows (chapter artwork and web links when appropriate), and we collected and published thorough show notes.
  • We recorded and edited the audio outside Anchor, and only published the mastered audio. It’s not perfect but it’s definitely less *rough*.
  • We bought a domain and we set up a website: KeepInTouch.fm.
  • We set up a Twitter account: @KeepInTouchFM.
  • We set up a vanity Google Alert for our show title and website.
  • Speaking of vanity, we created a Chartable.com account to keep an eye on how the show is doing in the charts (we were #7 in the Educational Technology category at one point!).
  • We transferred the ownership of the Apple Podcasts and Spotify feeds.
  • We set up the Stitcher and TuneIn feeds ourselves.
  • We integrated the podcast RSS feed with our new website: once we publish a show, a corresponding blog post gets created (almost) auto-magically.

Some of these enhancements will be the subject of future blog posts.

Things we learned

  • Apple Podcasts took about five days to process Anchor’s request to list our show.
  • Google Podcasts approved the show almost immediately but took about two weeks to actually make it available to listeners.
  • PocketCasts picked up the show immediately (most likely due to an integration between them an Anchor).
  • Anchor didn’t seem to get the show listed in TuneIn so we had to list add show manually.
  • Anchor doesn’t actually support all the podcasting networks.
  • Hosting the show’s website on GitHub pages was straightforward but did have some limitations. We forked Jekyll-Import to add support to the Jekyll RSS importer for Podcast RSS feeds, so we needed a home on GitHub. Hosting the website itself on GitHub made sense since Git enables a simple way to publish new episodes / blog posts.

Costs

  • Hosting and publishing the show: Free.
  • Hosting the website: Free.
  • Having a custom domain name: same cost as for any other type of website.
  • Podcast recording and editing tools: Free (not really, since we decided to use our own microphones and Digital Audio Workstation).
  • Time required to record, edit, and publish the show: 1 hour for the recording, 4 hours for editing (takes less and less each time), 30 minutes for packaging and publishing.

We are four episodes in, and we are having a lot of fun. I hope you will give our show a listen and let us know what you think. Keep in touch by subscribing on Apple Podcasts or Google Podcasts.

Sharing Information – My basic principles

As a team leader, I sometimes find myself in a situation where I have to communicate a decision to my team. Here are some key principles that I a guide myself by when I do this.

Do it as early as it makes sense

Timing is one of the most important things to me. If my team find out about the “news” from somewhere else, their trust in me is likely to be shaken. I am there to look after them and to be their voice in the conversations that have an impact on them. The sooner I can do that, the more I can help them.

Give a quick rundown of what will be covered

At the beginning of the discussion I try to clarify what I will talk about. This provides not just structure, but it may prevent questions that would be answered anyway. At the same time, it enables everyone to write down any questions that I may have overlooked in my FAQ.

Ideally, the agenda should be sent with the meeting invite, but sometimes this is just not possible.

Provide context

The reality is that not everyone reads their emails. If these emails contain business speak, are too lengthy, or come from a source that is too far away in the organisation, then it’s possible that people will just skim the message and not retain too much information.

The first part of the discussion should definitely include a quick recap of what has been going on, to bring everyone up to speed.

Stay honest, genuine, natural

I have seen so many people try to become somebody they are not during a presentation. Their pitch or tone would change. Their body language would be awkward. Their hand gestures would be unnatural. I try my hardest not to be one of these people.

Be yourself. Everyone else is taken.

I feel like being genuine helps reassure people that I am being honest. Even if the point of the meeting is to “sell” to the team something that they might not be too happy about, it’s still important that my tone is affiliative.

Give just the right amount of information

One very important aspect is finding the right balance between getting lost in the details (overwhelming the team with too much information) and being too vague about what the decision really entails.

Go through a FAQ

I try to think ahead of the questions that the team might have. Putting myself in their shoes is good for several reasons:

  • helps me prepare for the meeting
  • contributes to the decision of how much should be communicated
  • keeps me honest
  • prevents the piling up of answers such as “I don’t know”, and “I’ll get back to you”

Take questions

I am a strong believer in the power of an honest “I don’t know”. Realistically, if the truth is that I don’t know, then anything else I would say is BS. I respect my team too much to reply with a political, or vague answer. It’s also very likely that the answer will be important to me too.

Thank and close

When closing, it’s important to remind people that the communication channel is still open. Some people are introverts, or simply not comfortable asking questions in front of their peers. Others prefer to communicate in writing. Their voices are equally important therefore they need to know that they can follow-up with me.

Reboot

I need to put myself out there again. I am at crossroads with my career and I am hoping that writing about the decision making process and the context around it all will help me make the right decisions.

Relevant background

Instead of spending a lot of time and energy writing about my entire life, I will quickly summarise the last decade. Realistically, the last five years are the only relevant ones, so I will go into more detail as I approach current day.

2005 – 2009

About five years ago (2009), I was wrapping up my gig with Fronde Anywhere (a Mobile Banking & Finance pseudo-Start Up). I worked there between 2005 and 2009. My “consultant” title seemed to be the only generic term that could encapsulate all the roles I performed:

  • Software developer (Architecture, Mobile, Web, Systems Integration)
  • Trade Show presenter (e.g. CeBIT in Hannover)
  • RFP responder and integration partner
    • In New Zealand: ANZ, BNZ, KiwiBank, M-Co, Oracle, NZ Post, Telecom, Trade Me, Vodafone, Westpac
    • and abroad: BPI, Citi, Credit Agricole, Figaro, Sparkassen
  • Consultant and trainer (KiwiBank, CSC, Cap Gemini)
  • Marketing genius. Well not really, but I do take full credit for coming up with the Fronde name when Synergy International got rebranded

I soon realised that building software was just one facet of my technology persona. I also became aware that becoming involved with other businesses, and taking more product ownership was what I really wanted to do. Simply put, my view of my career and my priorities was somewhere along these lines:

  • short term: raise my profile as a Mobile technology person
  • medium term: build a popular Mobile solution that I would be proud of
  • long term: fill the gaps in my skills (leadership experience, design awareness, product mindedness)

The company I was working for was a good, comfortable place. However the challenges and the opportunities were not there so I had to make some changes. Without doubt, the most drastic change was starting Tmro. That was possible once I decided to reduce my work week from five to four days. I’ll dissect Tmro some other time, though…

2009 – …

The other change that I made was to take a job at Trade Me (2009). I joined as a back-end developer,  but at every occasion I was pushing the Mobile agenda. My .NET adventure was short lived. The contractor that had been hired to build an iPhone app was constantly asking for my support which was formalised within a few months. His contract wrapped up and I became the solo developer. We released the first version of the Trade Me for iPhone app sometime around Guy Fawkes (2010).

Trade Me Mobile

The iPhone app grew fast; so fast that within just a few months the Trade Me Mobile Team was formed, when Sam joined me as the Mobile Team Tester. Design was a separate entity altogether (It took a few years until the designers joined the developers and the testers to form cross skilled squads/teams as part of the larger Mobile Tribe/Team). The Mobile Team expansion continued by hiring two more iOS developers. We agreed to keep the developer to tester ratio as close as possible to 3 to 1. Before we knew it, we were investigating the possibility of building an Android app to repeat the success of the iPhone app. We did it ourselves, while building another two apps for a satellite business. With four apps to support we finally hired our own Mobile designer. This happened just in time for the first vertical app: the Trade Me Property for iOS app and the beginning of our bravest app: the Trade Me for iPad app. Things escalated quickly: in just a couple of years the app portfolio was growing. So was the team. What was shrinking was the time I was spending building solutions. I found myself doing more and more product advocacy (breaking business rules for the sake of making the apps better), arguing for resources, writing project plans and proposals, taking part in strategy meetings, and most importantly, managing people. The dream was coming true: my medium and long term plans were going great.

Crossroads

I did not anticipate was was about to happen next, but in hindsight it’s easy to understand how building a product suite, growing and managing an entire team, and representing these both internally and externally was going to transform me into a Product person. My job title said “Team Leader” but, in my heart, I saw myself as a Product Gatekeeper with a clear understanding that keeping the team happy was the first step towards delivering delight to our members. My vision was thus formed:

Only a happy team can build products that people love.

Despite attending WWDC (twice), Google Developer events, Rails camps, Webstock, UI/UX conferences, Agile barcamps, and more, I was quickly drifting away from being  just a software developer. My attention was constantly focused on the big picture rather than just the implementation detail. Most of the energy that used to be directed at learning new programming languages and development tools, was now invested in learning about what makes people tick and how to build products that people love.

Today’s turmoil

The Mobile Team is now large (approaching 30) and is made of a bunch of incredible people. There are five leaders who are amazing at representing their colleagues. Our skills cover Android, Design, iOS, Test, and Windows Phone.  We have outgrown Wellington (we are building a brand new team in Auckland). I sometimes wonder whether this is the largest Mobile Team in New Zealand. The products are more than just “relevant”.  Our official strategy documents require that “mobile is ingrained” in everything we do and this requirement is justified by fantastic stats. There’s a problem though: I want more. Currently, I find myself waking up in the middle of the night, taking notes and jotting down thoughts and ideas for a new Product that I have had in the back of my head for around four years now.

Reboot completed. Logging in…

I probably won’t be able to talk about the details of this Product for a while. I will write about the challenges that I come across and the decisions that I plan on making starting with the next post…

Engage, or not

You can do a number of things, too:

  • Nothing
  • Keep reading (subscribe to the RSS feed)
  • Submit feedback via my Twitter account or by using the contact page