Skip to content

Commit 8d8a9bb

Browse files
averikitschgrayside
averikitsch
and
grayside
authored
Add Cloud Run filesystem sample (GoogleCloudPlatform#6723)
* draft * initial * Fuse info * Clean up * remove unused files * fix * Add secrets * fix header * lint * update kokoro * remove alpha install * clean * debug * comments * add fuse instructions * Update tests * update headers * fix platform flag * respond to comments * remove debug * add region tags * fix test * fix test * lint * respond to comments Co-authored-by: Adam Ross <[email protected]>
1 parent c5bc64a commit 8d8a9bb

File tree

12 files changed

+506
-0
lines changed

12 files changed

+506
-0
lines changed

‎run/filesystem/Dockerfile‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Copyright 2021 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# [START cloudrun_fs_dockerfile]
16+
# Use the official lightweight Python image.
17+
# https://hub.docker.com/_/python
18+
FROM python:3.9-slim
19+
20+
# Install system dependencies
21+
RUN apt-get update -y && apt-get install -y \
22+
tini \
23+
nfs-common \
24+
&& apt-get clean
25+
26+
# Set fallback mount directory
27+
ENV MNT_DIR /mnt/nfs/filestore
28+
29+
# Copy local code to the container image.
30+
ENV APP_HOME /app
31+
WORKDIR $APP_HOME
32+
COPY . ./
33+
34+
# Install production dependencies.
35+
RUN pip install -r requirements.txt
36+
37+
# Ensure the script is executable
38+
RUN chmod +x /app/run.sh
39+
40+
# Use tini to manage zombie processes and signal forwarding
41+
# https://github.com/krallin/tini
42+
ENTRYPOINT ["/usr/bin/tini", "--"]
43+
44+
# Pass the startup script as arguments to tini
45+
CMD ["/app/run.sh"]
46+
# [END cloudrun_fs_dockerfile]

‎run/filesystem/README.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Cloud Run File System Sample
2+
3+
This sample shows how to create a service that mounts a Filestore
4+
instance as a network file system.
5+
6+
## Tutorials
7+
See our [Using Filestore with Cloud Run tutorial](https://cloud.google.com/run/docs/tutorials/network-filesystems) or [Using Cloud Storage FUSE with Cloud Run tutorial](https://cloud.google.com/run/docs/tutorials/network-filesystems-fuse) for instructions for setting up and deploying this sample application.
8+
9+
[create]: https://cloud.google.com/storage/docs/creating-buckets
10+
[fuse]: https://cloud.google.com/storage/docs/gcs-fuse
11+
[git]: https://github.com/GoogleCloudPlatform/gcsfuse
12+
[auth]: https://cloud.google.com/artifact-registry/docs/docker/authentication

‎run/filesystem/cloudbuild.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Copyright 2021 Google LLC
2+
# Licensed under the Apache License, Version 2.0 (the "License");
3+
# you may not use this file except in compliance with the License.
4+
# You may obtain a copy of the License at
5+
# http://www.apache.org/licenses/LICENSE-2.0
6+
# Unless required by applicable law or agreed to in writing, software
7+
# distributed under the License is distributed on an "AS IS" BASIS,
8+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9+
# See the License for the specific language governing permissions and
10+
# limitations under the License.
11+
12+
steps:
13+
- id: "Deploy to Cloud Run"
14+
name: "gcr.io/cloud-builders/gcloud:latest"
15+
entrypoint: /bin/bash
16+
args:
17+
- "-c"
18+
- |
19+
gcloud alpha run deploy ${_SERVICE_NAME} \
20+
--source . \
21+
--execution-environment gen2 \
22+
--vpc-connector ${_CONNECTOR} \
23+
--update-env-vars FILESTORE_IP_ADDRESS=${_FILESTORE_IP_ADDRESS},FILE_SHARE_NAME=${_FILESHARE} \
24+
--region ${_DEPLOY_REGION} \
25+
--no-allow-unauthenticated
26+
27+
substitutions:
28+
_SERVICE_NAME: filesystem
29+
_DEPLOY_REGION: us-central1
30+
_FILESHARE: vol1
31+
_CONNECTOR: my-run-connector
32+
_FILESTORE_IP_ADDRESS: 0.0.0.0

‎run/filesystem/e2e_test.py‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Copyright 2021 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# This sample creates a secure two-service application running on Cloud Run.
16+
# This test builds and deploys the two secure services
17+
# to test that they interact properly together.
18+
19+
import datetime
20+
import os
21+
import subprocess
22+
from urllib import request
23+
import uuid
24+
25+
import pytest
26+
27+
# Unique suffix to create distinct service names
28+
SUFFIX = uuid.uuid4().hex
29+
PROJECT = os.environ['GOOGLE_CLOUD_PROJECT']
30+
31+
32+
@pytest.fixture
33+
def deployed_service():
34+
# Deploy image to Cloud Run
35+
service_name = f'filesystem-{SUFFIX}'
36+
connector = os.environ['CONNECTOR']
37+
ip_address = os.environ['IP_ADDRESS']
38+
39+
subprocess.run(
40+
[
41+
'gcloud',
42+
'builds',
43+
'submit',
44+
'--config',
45+
'cloudbuild.yaml',
46+
'--project',
47+
PROJECT,
48+
f'--substitutions=_SERVICE_NAME={service_name},_FILESTORE_IP_ADDRESS={ip_address},_CONNECTOR={connector}'
49+
],
50+
check=True,
51+
)
52+
53+
yield service_name
54+
55+
subprocess.run(
56+
[
57+
'gcloud',
58+
'run',
59+
'services',
60+
'delete',
61+
service_name,
62+
'--region=us-central1',
63+
'--platform=managed',
64+
'--quiet',
65+
'--project',
66+
PROJECT,
67+
],
68+
check=True,
69+
)
70+
71+
72+
@pytest.fixture
73+
def service_url_auth_token(deployed_service):
74+
# Get Cloud Run service URL and auth token
75+
service_url = (
76+
subprocess.run(
77+
[
78+
'gcloud',
79+
'run',
80+
'services',
81+
'describe',
82+
deployed_service,
83+
'--region=us-central1',
84+
'--platform=managed',
85+
'--format=value(status.url)',
86+
'--project',
87+
PROJECT,
88+
],
89+
stdout=subprocess.PIPE,
90+
check=True,
91+
)
92+
.stdout.strip()
93+
.decode()
94+
)
95+
auth_token = (
96+
subprocess.run(
97+
['gcloud', 'auth', 'print-identity-token'],
98+
stdout=subprocess.PIPE,
99+
check=True,
100+
)
101+
.stdout.strip()
102+
.decode()
103+
)
104+
105+
yield service_url, auth_token
106+
107+
# no deletion needed
108+
109+
110+
def test_end_to_end(service_url_auth_token):
111+
service_url, auth_token = service_url_auth_token
112+
# Non mnt directory
113+
req = request.Request(
114+
service_url, headers={'Authorization': f'Bearer {auth_token}'}
115+
)
116+
response = request.urlopen(req)
117+
assert response.status == 200
118+
119+
# Mnt directory
120+
mnt_url = f'{service_url}/mnt/nfs/filestore'
121+
req = request.Request(
122+
mnt_url, headers={'Authorization': f'Bearer {auth_token}'}
123+
)
124+
response = request.urlopen(req)
125+
assert response.status == 200
126+
127+
date = datetime.datetime.utcnow()
128+
body = response.read()
129+
assert '{dt:%a}-{dt:%b}-{dt:%d}-{dt:%H}:{dt:%M}-{dt:%Y}'.format(dt=date) in body.decode()

‎run/filesystem/gcsfuse.Dockerfile‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Copyright 2021 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# [START cloudrun_fuse_dockerfile]
16+
# Use the official lightweight Python image.
17+
# https://hub.docker.com/_/python
18+
FROM python:3.9-buster
19+
20+
# Install system dependencies
21+
RUN set -e; \
22+
apt-get update -y && apt-get install -y \
23+
tini \
24+
lsb-release; \
25+
gcsFuseRepo=gcsfuse-`lsb_release -c -s`; \
26+
echo "deb http://packages.cloud.google.com/apt $gcsFuseRepo main" | \
27+
tee /etc/apt/sources.list.d/gcsfuse.list; \
28+
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | \
29+
apt-key add -; \
30+
apt-get update; \
31+
apt-get install -y gcsfuse \
32+
&& apt-get clean
33+
34+
# Set fallback mount directory
35+
ENV MNT_DIR /mnt/gcs
36+
37+
# Copy local code to the container image.
38+
ENV APP_HOME /app
39+
WORKDIR $APP_HOME
40+
COPY . ./
41+
42+
# Install production dependencies.
43+
RUN pip install -r requirements.txt
44+
45+
# Ensure the script is executable
46+
RUN chmod +x /app/gcsfuse_run.sh
47+
48+
# Use tini to manage zombie processes and signal forwarding
49+
# https://github.com/krallin/tini
50+
ENTRYPOINT ["/usr/bin/tini", "--"]
51+
52+
# Pass the startup script as arguments to Tini
53+
CMD ["/app/gcsfuse_run.sh"]
54+
# [END cloudrun_fuse_dockerfile]

‎run/filesystem/gcsfuse_run.sh‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/usr/bin/env bash
2+
# Copyright 2021 Google LLC
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
# [START cloudrun_fuse_script]
16+
set -eo pipefail
17+
18+
# Create mount directory for service
19+
mkdir -p $MNT_DIR
20+
21+
echo "Mounting GCS Fuse."
22+
gcsfuse --debug_gcs --debug_fuse $BUCKET $MNT_DIR
23+
echo "Mounting completed."
24+
25+
# Run the web service on container startup. Here we use the gunicorn
26+
# webserver, with one worker process and 8 threads.
27+
# For environments with multiple CPU cores, increase the number of workers
28+
# to be equal to the cores available.
29+
# Timeout is set to 0 to disable the timeouts of the workers to allow Cloud Run to handle instance scaling.
30+
exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app &
31+
32+
# Exit immediately when one of the background processes terminate.
33+
wait -n
34+
# [END cloudrun_fuse_script]

0 commit comments

Comments
 (0)