Inconsistent memcache lookups from different Google AppEngine modules of the same app

Question

We have come across a sporadic inconsistency when doing memcache lookups from different modules (aka "services") within a single AppEngine app (Python). The scenario is as such:

• Module "portal" creates and later increments integer values for a certain memcache key:

  @ndb.tasklet
  def increment_total_async(meter):
      total = yield memcache.Client().incr_async(
          key="some-key",
          namespace=TOTALS_NAMESPACE,
          delta=meter.quantity)
  
      if total is not None:
          raise ndb.Return(total)
  
      raise ndb.Return(None)
  

• Module "dashboard" later looks up the current value using the same key and namespace:

  @ndb.transactional_tasklet
  def get_total_async(subscription, metric):
      total = memcache.get(
          key="some-key",
          namespace=TOTALS_NAMESPACE)
  
      if total is not None:
          raise ndb.Return(total)
  
      raise ndb.Return(None)
  

Interestingly, get_total_async sometimes (rarely) returns an integer value of 0 (not None) when called from the "dashboard" module, while when called from the "portal" module it returns the (correct) value >0.

This is somewhat surprising and makes us believe that there may be a race condition in the memcache service - possibly only when one looks up the same key from different modules (now called "services" by Google).

Has anyone come across a similar phenomenon? This is a "Heisenbug", i.e., difficult to reproduce.