Skip to content

Commit

Permalink
Finished Processing and Added Client
Browse files Browse the repository at this point in the history
  • Loading branch information
@domp47
domp47 committed Apr 23, 2019
1 parent b987485 commit bb83d3e
Show file tree
Hide file tree
Showing 2 changed files with 378 additions and 54 deletions.
301 changes: 254 additions & 47 deletions docs/installation.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,231 @@

## Client Server Installation

The following has been tested on Ubuntu 18.04 LTS, the system should work on any modern Unix like OS that is supported by dotnet. Run the following commands to install the dependencies required.

```bash
#Dotnet 2.2
wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo add-apt-repository universe
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.2
rm packages-microsoft-prod.deb

#PostgreSQL
sudo apt-get install postgresql

#Nodejs & npm
sudo apt install nodejs npm

#Python dependencies
sudo apt-get install python3-dev python-virtualenv
```

#### Set up the client database
- login to the database with `sudo -i -u postgres psql` then set up the database and users. Replace `clientserver`, `clientuser`, and `password` with your desired database name, database user, and password, and then update the `appsettings.json` to reflect what you set them to.

```
CREATE DATABASE clientserver;
CREATE USER clientuser WITH ENCRYPTED PASSWORD 'password';
GRANT ALL PRIVILEGES ON DATABASE clientserver TO clientuser;
```

- Setup the schema with the following
```
dotnet ef migrations add InitialCreate
dotnet ef database update
```

#### Set up scrubbing script
- Locate the scrubbing folder inside the client code folder then create a virtual environment and install the the dependencies in it.
```
virtualenv -p python3 env
source env/bin/activate
pip install -r requirements.txt
```

#### Set up webserver
- Start by installing NGINX, `sudo apt-get install nginx`.
- Run `dotnet publish -c Release` inside the ClientServer folder, this will create a release inside `./bin/Release/netcoreapp2.2/publish/`. Now copy that `publish/` folder to `/var/www/` and name the folder `ClientServer`.
- Set up the NGINX server as a reverse proxy, create a file named `/etc/nginx/sites-available/default` with the following text (`default` can be changed to anything you want).
```
server {
listen 80;
server_name domainname.com www.domainname.com;
location / {
proxy_pass http://localhost:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```

- This sets up the server for HTTP only. Before NGINX implements the configuration you need to create a link of this file into the enabled folder with the following command `sudo ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/`. Restart NGINX to implement the changes made with `sudo nginx -s reload`.

- Now setup the kestrel server for dotnet, create a file called `/etc/systemd/system/kestrel-ClientServer.service` with the following text. If you didn't put the application in `/var/www/` then replace it with where ever you put it, but make sure to change the folder ownership and permissions.
```
[Unit]
Description=Hook Client
[Service]
WorkingDirectory=/var/www/ClientServer
ExecStart=/usr/bin/dotnet /var/www/ClientServer/ClientServer.dll
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=hook-clientserver
User=www-data
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
TimeoutStopSec=90
[Install]
WantedBy=multi-user.target
```
- Now enable and start the service.
```
sudo systemctl enable kestrel-ClientServer.service
sudo systemctl start kestrel-ClientServer.service
sudo systemctl status kestrel-ClientServer.service
```

- Make sure that the status says loaded and active, press `q` to quit the log.

- To set up the proxy create a file called `/etc/nginx/proxy.conf` with the following text
```
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffers 32 4k;
```

- Set up the `/etc/nginx/nginx.conf` file by deleting the contents and replacing it with the following, replace the ssl certificate and key file with your own if you already have a CA signed cert for the server being used.

```
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
include /etc/nginx/proxy.conf;
limit_req_zone $binary_remote_addr zone=one:10m rate=5r/s;
server_tokens off;
sendfile on;
keepalive_timeout 1000; # Adjust to the lowest possible value that makes sense for your use case.
client_body_timeout 1000; client_header_timeout 1000; send_timeout 1000;
upstream hellomvc{
server localhost:5000;
}
server {
listen *:80;
add_header Strict-Transport-Security max-age=15768000;
return 301 https://$host$request_uri;
}
server {
listen *:443 ssl;
server_name 35.227.54.68;
ssl_certificate /etc/ssl/certs/ClientServer.crt;
ssl_certificate_key /etc/ssl/private/ClientServer.key;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on; #ensure your cert is capable
ssl_stapling_verify on; #ensure your cert is capable
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
#Redirects all traffic
location / {
proxy_pass http://hellomvc;
limit_req zone=one burst=10 nodelay;
}
}
}
```

#### Creating a CA signed cert

- If you already have a domain name but don't have a CA signed cert for the server do the following to create and install one.

```
sudo add-apt-repository ppa:certbot/certbot
sudo apt install python-certbot-nginx
```

- Make sure you have the correct `server_name domainname.com www.domainname.com` in the `/etc/nginx/sites-available/default` file before continuing.

```
sudo certbot --nginx -d domainname.com -d www.domainname.com
```

- Follow the steps in the script and then it should automatically set the certs in the `default` file, the certificate expires every 90 days so to automate the renewel of it run `sudo certbot renew --dry-run`.

## Processing Server Installation

The following has been tested on Ubuntu 18.04 LTS but should work as well on any system running a minimum python version of 3.6.7, and Apache 2.4.

### PostgreSQL

- Start by installing postgreSQL
```
sudo apt-get update
sudo apt-get install postgresql
```

- Change the default postgres password to something secure with `sudo passwd postgres`
- Now create a database and user for the processing server. Replace `dbName` and `userName` with the desired database name and username respectively.
```bash
sudo -i -u postgres
createuser userName -P --interactive #superuser - n, create db - y, create role -y
createdb dbName
sudo adduser userName #leave everything blank except for password if you don't care
```
- Now that postgreSQL is installed log out of the `postgres` user and run `sudo -i -u userName pg_restore --no-owner -d dbName /path/to/dump.dmp` to restore the database. If for any reason you need to back up the schema run the following command `pg_dump -Fc dbName > dump.dmp`

### Source Code

- Copy the the `app.wsgi` file, `requirements.txt` file and `Src/` folder to a desired running directory

- Run the `setup.py` configuration setup file inside the `Src/` folder and follow the instructions in the script. **NOTE:** The path that is specified for the Queue and Results must already exist or else it the processing server will crash.
- Run the `setup.py` configuration setup file inside the `Src/` folder and follow the instructions in the script. **NOTE:** The path that is specified for the Queue and Results must already exist or else the processing server will crash. Also the folder you specify must have r/w/x permissions for the `www-data` user, this can be done with `setfacl -m u:www-data:rwx aFolder`.

- Open a terminal inside the `Src/` folder and enter the following commands
- `python3 -m venv ./env`
- `source env/bin/activate`
- `pip install -r requirements.txt`
```
sudo apt-get install python-virtualenv
virtualenv -p python3 ./env
source env/bin/activate
pip install -r ../requirements.txt
```
- The dependencies should now be satisfied and you can test if they are installed correctly by running `python __init__.py`, if it runs correctly you can continue.

- Now that the code is running it is time to install the webserver to run it on. It is also possible to use NGINX but it has not been tested. To install apache and WSGI run the following commands:
Expand Down Expand Up @@ -51,34 +260,14 @@ Header always set X-Content-Type-Options nosniff
SSLOptions +StrictRequire
```

- Now create a file `/etc/apache2/sites-available/app.conf` with the text
- Now create a file `/etc/apache2/sites-available/app.conf` with the text below replacing all the paths with where the source code was copied to.
```
<IfModule mod_ssl.c>
<VirtualHost *:port>
# The ServerName directive sets the request scheme, hostname and port that
# the server uses to identify itself. This is used when creating
# redirection URLs. In the context of virtual hosts, the ServerName
# specifies what hostname must appear in the request's Host: header to
# match this virtual host. For the default virtual host (this file) this
# value is not decisive as it is used as a last resort host regardless.
# However, you must set it for any further virtual host explicitly.
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
ErrorLog /home/4F00/chloroplasts/error.log
LogLevel info
CustomLog /home/4F00/chloroplasts/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
SSLCertificateFile /etc/ssl/certs/apache-selfsigned.crt
SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key
ServerName ipAddress
Expand Down Expand Up @@ -115,7 +304,7 @@ sudo a2enconf ssl-params
sudo systemctl restart apache2
```

- This should enable the webserver with WSGI and TLS. The certificate expires one year from the day you create it, so it will have to be renewed every year.
- This should enable the webserver with WSGI and TLS. The certificate expires one year from the day you create it, so it will have to be renewed manually every year.

#### If the server has a FQDN then you can use a CA signed cert

Expand All @@ -126,30 +315,48 @@ sudo apt-get update
sudo apt-get install python-certbot-apache
```

- Then create a file `/etc/apache2/sites-available/app.conf` with the text
- Then create a file `/etc/apache2/sites-available/app.conf` with the text below replacing all the paths with where the source code was copied to and `dn.tld` with your FQDN.
```
<VirtualHost *:port>
ErrorLog /home/4F00/chloroplasts/error.log
LogLevel info
CustomLog /home/4F00/chloroplasts/access.log combined
ServerName dn.tld
ErrorLog /home/4F00/chloroplasts/error.log
LogLevel info
CustomLog /home/4F00/chloroplasts/access.log combined
Alias /static /home/4F00/chloroplasts/Src/static
<Directory /home/4F00/chloroplasts/Src/static>
<RequireAll>
Require all granted
</RequireAll>
</Directory>
Alias /static /home/4F00/chloroplasts/Src/static
<Directory /home/4F00/chloroplasts/Src/static>
<RequireAll>
<Directory /home/4F00/chloroplasts>
<Files app.wsgi>
<RequireAll>
Require all granted
</RequireAll>
</Directory>
<Directory /home/4F00/chloroplasts>
<Files app.wsgi>
<RequireAll>
Require all granted
</RequireAll>
</Files>
</Directory>
WSGIDaemonProcess myproject python-path=/home/4F00/chloroplasts python-home=/home/4F00/chloroplasts/Src/env
WSGIProcessGroup myproject
WSGIScriptAlias / /home/4F00/chloroplasts/app.wsgi
</Files>
</Directory>
WSGIDaemonProcess myproject python-path=/home/4F00/chloroplasts python-home=/home/4F00/chloroplasts/Src/env
WSGIProcessGroup myproject
WSGIScriptAlias / /home/4F00/chloroplasts/app.wsgi
</VirtualHost>
```
```

- Now run the command `sudo certbot --apache -d dn.tld -d www.dn.tld` and follow the instructions in the script. Everything should be set up automatically by Let's Encrypt, you can run a `apachectl configtest` to test the configuration of Apache. If the configtest fails but certbot didn't throw any errors verify that the following lines were inserted into your `app.conf` file.
```
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/www.facialstats.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.facialstats.com/privkey.pem
```

- If the configtest is successful then run the following commands to enable the site and restart Apache.
```
sudo a2ensite app
sudo systemctl restart apache2
```

- This is optional but recommended as the certificate expires every 90 days.
- Open up a terminal and enter `sudo crontab -e` then add the following line to the end of your cronjob file `15 3 * * * /usr/bin/certbot renew --quiet`. This renews the cert if it is set to expire in the next month.
Loading

0 comments on commit bb83d3e

Please sign in to comment.