Samples for migrating from Python 2.7 runtime to Python 3.7 (GoogleCl…

…oudPlatform#3656)

* Initial commit for storage sample

* Added Python 3.7 version

* Starting urlfetch samples

* requests samples work in 2.7 and 3.7 (gcloud deploy app app3.yaml for the Python 3.7 version)

* urlfetch async sample Working in Python 3 App Engine

* Working in Python 2 App Engine, too

* Added storage test for migration

* Use standard environment variable name for bucket

* Added tests

* WIP on urlfetch authentication

* Added new authenticated replacement for urlfetch

* Added a test

* Incoming app identity verification sample

* Added test

* Added READMEs

* Update README.md

* Update main.py

* Update main.py

* Update main.py

* Include exception in log message

* Fixed missing newline at EOF

* Removed unused import

* Removed unneeded import

* Minor lint cleanup

* Removed unneeded import

* Added EOF newline

* Lint fix

* Removed unused import requests

* Removed unused import

* Tell linter to ignore needed module import order

* Reordered imports

* Adding requirements-test to samples

* Linting directive added

* Replaced test requirements with Python 2.7 compatible one

* Another missing requirements-test.txt added

* More requirements file updates

* Typo fixes

* Migration cleanup for automated tests

* Adjusted tests for Python 2.7

* Make lint happier

* Add requests adapter

* Needed stub to run in test environment

* Trying to get test to run in this environment

* WIP

* WIP

* WIP

appengine/standard/migration/incoming/README.md

@@ -0,0 +1,16 @@
+## App Engine to App Engine Request Sample
+
+This sample application shows how an App Engine for Python 2.7 app can verify
+the caller's identity for a request from an App Engine for Python 2.7 or
+App Engine for Python 3.7 app.
+
+Requests from an App Engine for Python 2.7 app that use `urlfetch` have
+a trustworthy `X-Appengine-Inbound-Appid` header that can be used to verify
+the calling app's identity.
+
+Requests from an App Engine for Python 3.7 app that include an Authorization
+header with an ID token for the calling app's default service account can
+be used to verify those calling apps' identities.
+
+The appengine/standard_python37/migration/urlfetch sample app can be used
+to make calls to this app with a valid Authorization header.

appengine/standard/migration/incoming/app.yaml

@@ -0,0 +1,25 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+runtime: python27
+threadsafe: yes
+api_version: 1
+
+libraries:
+- name: ssl
+  version: latest
+
+handlers:
+- url: .*
+  script: main.app

appengine/standard/migration/incoming/appengine_config.py

@@ -0,0 +1,20 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# [START vendor]
+from google.appengine.ext import vendor
+
+# Add any libraries installed in the "lib" folder.
+vendor.add('lib')
+# [END vendor]

appengine/standard/migration/incoming/main.py

@@ -0,0 +1,84 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Authenticate requests coming from other App Engine instances.
+"""
+
+# [START gae_python_app_identity_incoming]
+from google.oauth2 import id_token
+from google.auth.transport import requests
+
+import logging
+import webapp2
+
+
+def get_app_id(request):
+    # Requests from App Engine Standard for Python 2.7 will include a
+    # trustworthy X-Appengine-Inbound-Appid. Other requests won't have
+    # that header, as the App Engine runtime will strip it out
+    incoming_app_id = request.headers.get(
+        'X-Appengine-Inbound-Appid', None)
+    if incoming_app_id is not None:
+        return incoming_app_id
+
+    # Other App Engine apps can get an ID token for the App Engine default
+    # service account, which will identify the application ID. They will
+    # have to include at token in an Authorization header to be recognized
+    # by this method.
+    auth_header = request.headers.get('Authorization', None)
+    if auth_header is None:
+        return None
+
+    # The auth_header must be in the form Authorization: Bearer token.
+    bearer, token = auth_header.split()
+    if bearer.lower() != 'bearer':
+        return None
+
+    try:
+        info = id_token.verify_oauth2_token(token, requests.Request())
+        service_account_email = info['email']
+        incoming_app_id, domain = service_account_email.split('@')
+        if domain != 'appspot.gserviceaccount.com':  # Not App Engine svc acct
+            return None
+        else:
+            return incoming_app_id
+    except Exception as e:
+        # report or log if desired, as here:
+        logging.warning('Request has bad OAuth2 id token: {}'.format(e))
+        return None
+
+
+class MainPage(webapp2.RequestHandler):
+    allowed_app_ids = [
+        'other-app-id',
+        'other-app-id-2'
+    ]
+
+    def get(self):
+        incoming_app_id = get_app_id(self.request)
+
+        if incoming_app_id is None:
+            self.abort(403)
+
+        if incoming_app_id not in self.allowed_app_ids:
+            self.abort(403)
+
+        self.response.write('This is a protected page.')
+
+
+app = webapp2.WSGIApplication([
+    ('/', MainPage)
+], debug=True)
+# [END gae_python_app_identity_incoming]

appengine/standard/migration/incoming/main_test.py

@@ -0,0 +1,27 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import webtest
+
+import main
+
+
+def test_get():
+    app = webtest.TestApp(main.app)
+
+    try:
+        response = app.get('/')
+        assert response.status_int == 403
+    except webtest.app.AppError as e:
+        assert '403 Forbidden' in str(e)

appengine/standard/migration/incoming/requirements-test.txt

@@ -0,0 +1,2 @@
+pytest>=4.6.10
+webtest>=2.0.35

appengine/standard/migration/incoming/requirements.txt

@@ -0,0 +1,2 @@
+google-auth>=1.14.1
+requests>=2.23.0

appengine/standard/migration/storage/README.md

@@ -0,0 +1,16 @@
+## App Engine cloudstorage library Replacement
+
+The runtime for App Engine standard for Python 2.7 includes the `cloudstorage`
+library, which is used to store and retrieve blobs. The sample in this
+directory shows how to do the same operations using Python libraries that
+work in either App Engine standard for Python runtime, version 2.7 or 3.7.
+The sample code is the same for each environment.
+
+To deploy and run this sample in App Engine standard for Python 2.7:
+
+    pip install -t lib -r requirements.txt
+    gcloud app deploy
+
+To deploy and run this sample in App Engine standard for Python 3.7:
+
+    gcloud app deploy app3.yaml

appengine/standard/migration/storage/app.yaml

@@ -0,0 +1,15 @@
+runtime: python27
+api_version: 1
+threadsafe: yes
+
+env_variables:
+    CLOUD_STORAGE_BUCKET: "REPLACE THIS WITH YOUR EXISTING BUCKET NAME"
+    BLOB_NAME: "my-demo-blob"
+
+libraries:
+  - name: ssl
+    version: latest
+
+handlers:
+  - url: /.*
+    script: main.app

appengine/standard/migration/storage/app3.yaml

@@ -0,0 +1,5 @@
+runtime: python37
+
+env_variables:
+    CLOUD_STORAGE_BUCKET: "REPLACE THIS WITH YOUR EXISTING BUCKET NAME"
+    BLOB_NAME: "my-demo-blob"

appengine/standard/migration/storage/appengine_config.py

@@ -0,0 +1,4 @@
+from google.appengine.ext import vendor
+
+# Add any libraries installed in the "lib" folder.
+vendor.add('lib')

appengine/standard/migration/storage/main.py

@@ -0,0 +1,60 @@
+# Copyright 2020 Google, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from flask import Flask, make_response
+import os
+
+from google.cloud import storage
+
+
+app = Flask(__name__)
+
+
+@app.route('/', methods=['GET'])
+def get():
+    bucket_name = os.environ['CLOUD_STORAGE_BUCKET']
+    blob_name = os.environ.get('BLOB_NAME', 'storage-migration-test-blob')
+
+    client = storage.Client()
+    bucket = client.bucket(bucket_name)
+    blob = bucket.blob(blob_name)
+
+    response_text = ''
+
+    text_to_store = b'abcde\n' + b'f'*1024*4 + b'\n'
+    blob.upload_from_string(text_to_store)
+    response_text += 'Stored text in a blob.\n\n'
+
+    stored_contents = blob.download_as_string()
+    if stored_contents == text_to_store:
+        response_text += 'Downloaded text matches uploaded text.\n\n'
+    else:
+        response_text += 'Downloaded text DOES NOT MATCH uploaded text!\n\n'
+
+    response_text += 'Blobs in the bucket:\n'
+    for blob in client.list_blobs(bucket_name):
+        response_text += '    ' + blob.id + '\n'
+    response_text += '\n'
+
+    bucket.delete_blob(blob_name)
+    response_text += 'Blob ' + blob_name + ' deleted.\n'
+
+    response = make_response(response_text, 200)
+    response.mimetype = 'text/plain'
+    return response
+
+
+if __name__ == '__main__':
+    # This is used when running locally.
+    app.run(host='127.0.0.1', port=8080, debug=True)

appengine/standard/migration/storage/main_test.py

@@ -0,0 +1,30 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import main
+import os
+
+
+def test_index():
+    main.app.testing = True
+    client = main.app.test_client()
+
+    r = client.get('/')
+    assert r.status_code == 200
+    assert 'Downloaded text matches uploaded text' in r.data.decode('utf-8')
+
+    bucket_name = os.environ['CLOUD_STORAGE_BUCKET']
+    blob_name = os.environ.get('BLOB_NAME', 'storage-migration-test-blob')
+    assert '    {}/{}'.format(bucket_name, blob_name) in r.data.decode('utf-8')
+    assert 'Blob {} deleted.'.format(blob_name) in r.data.decode('utf-8')

appengine/standard/migration/storage/requirements-test.txt

@@ -0,0 +1 @@
+pytest>=4.6.10

appengine/standard/migration/storage/requirements.txt

@@ -0,0 +1,2 @@
+google-cloud-storage==1.25.0
+Flask==1.1.1

appengine/standard/migration/urlfetch/async/README.md

@@ -0,0 +1,23 @@
+## App Engine async urlfetch Replacement
+
+The runtime for App Engine standard for Python 2.7 includes the `urlfetch`
+library, which is used to make HTTP(S) requests. There are several related
+capabilities provided by that library:
+
+* Straightforward web requests
+* Asynchronous web requests
+* Platform authenticated web requests to other App Engine apps
+
+The sample in this directory provides a way to make asynchronous web requests
+using only generally available Python libraries that work in either App Engine
+standard for Python runtime, version 2.7 or 3.7. The sample code is the same
+for each environment.
+
+To deploy and run this sample in App Engine standard for Python 2.7:
+
+    pip install -t lib -r requirements.txt
+    gcloud app deploy
+
+To deploy and run this sample in App Engine standard for Python 3.7:
+
+    gcloud app deploy app3.yaml

appengine/standard/migration/urlfetch/async/app.yaml

@@ -0,0 +1,7 @@
+runtime: python27
+api_version: 1
+threadsafe: yes
+
+handlers:
+- url: .*
+  script: main.app

appengine/standard/migration/urlfetch/async/app3.yaml

@@ -0,0 +1 @@
+runtime: python37

appengine/standard/migration/urlfetch/async/appengine_config.py

@@ -0,0 +1,32 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.appengine.api import apiproxy_stub_map
+from google.appengine.api import urlfetch_stub
+from google.appengine.ext import vendor
+
+# Add any libraries installed in the "lib" folder.
+vendor.add('lib')
+
+import requests_toolbelt.adapters.appengine  # noqa: E402
+
+# Use the App Engine Requests adapter. This makes sure that Requests uses
+# URLFetch.
+requests_toolbelt.adapters.appengine.monkeypatch()
+
+apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
+apiproxy_stub_map.apiproxy.RegisterStub(
+    'urlfetch',
+    urlfetch_stub.URLFetchServiceStub()
+)