How to store database credentials in Google App Engine Standard?

Question

I'm experimenting with hosting a Django application on Google App Engine Standard and to do that I followed the tutorial Running Django on App Engine Standard Environment. That tutorial completely ignores the problem of safely handling credentials (and it sorts of points to just storing them in your code repo... sigh).

When searching for how to store credentials there's a lot of articles about Google Cloud Key Management Service, but I don't really need that. I'm ok with the credentials being stored at GCP unencrypted (it's for the GCP database anyway). What I need is a way for my application to pick up that configuration automatically.

My app can either get environment variables or a .env file that contains this information. How do I get the credentials and other configs to be present like that in my Google App Engine Standard instance?

Answer 1

The answer to this has driven me nuts for a while, but I finally have the answer. It uses Memcached, so it's fast in production.

1. In GCP, go to APIs & Services, and enable the Google Datastore API.
2. Then setup a Service Account and authenticate according to this tutorial: https://cloud.google.com/docs/authentication/getting-started
3. In your local envrionment, pip install google-cloud-datastore.
4. Then you'll want to create a module in the same folder as settings.py called something like datastore.py:

Here is my datastore.py:

from google.cloud import datastore
from django.core.cache import cache


class DataStoreClient():
    def __init__(self):
        self.client = datastore.Client()

    def get(self, property):
        try:
            cache_key = 'env-' + property
            result = cache.get(cache_key)

            if not result:
                key = self.client.key('environment_variables', property)
                result = self.client.get(key)
                cache.set(cache_key, result, 86400)

            return result['Value']

        except TypeError:
            print(
                "{} is not a property in Cloud Datastore".format(property) +
                "We are creating one for you.  Go to Cloud Datastore to set a value."
            )
            entity = datastore.Entity(key=key)
            entity.update({
                'Value': 'NOT_SET'}
            )
            self.client.put(entity)

Then in the settings, you just call for keys like this:

from your_app.datastore import DataStoreClient

datastore_client = DataStoreClient()

SECRET_KEY = datastore_client.get('SECRET_KEY')

When you run python manage.py runserver for the first time, the system will populate NOT_SET values in Google DataStore. From here, you can go to GCP and change those keys on datastore.

This solution is nice in that, once there is a value, it will retrieve it for you.

If you really want to get crazy (as I have), you could set up a script that includes all your environment variables locally, encrypts them with KMS, uploads them, then write the DataStoreClient to only download and decrypt.