-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
0.5.0 to master
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
name: Unit tests on Node.js | ||
|
||
on: | ||
push: | ||
branches: [ develop ] | ||
pull_request: | ||
branches: [ develop ] | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-latest | ||
|
||
strategy: | ||
matrix: | ||
node-version: [14.10] | ||
|
||
steps: | ||
- uses: actions/checkout@v2 | ||
- name: Use Node.js ${{ matrix.node-version }} | ||
uses: actions/setup-node@v1 | ||
with: | ||
node-version: ${{ matrix.node-version }} | ||
- name: Cache Node.js modules | ||
uses: actions/cache@v2 | ||
with: | ||
path: ~/.npm | ||
key: ${{ runner.OS }}-node-${{ hashFiles('**/yarn.lock') }} | ||
restore-keys: | | ||
${{ runner.OS }}-node- | ||
${{ runner.OS }}- | ||
- name: Install dependencies | ||
working-directory: ./ui | ||
run: npm install | ||
- name: Run tests | ||
working-directory: ./ui | ||
run: npm test -- --no-watch --no-progress --browsers=ChromeHeadlessCI |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
name: Unit tests on Python test | ||
|
||
on: | ||
push: | ||
branches: [ develop ] | ||
pull_request: | ||
branches: [ develop ] | ||
jobs: | ||
build: | ||
|
||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
python-version: [3.7] | ||
|
||
steps: | ||
- uses: actions/checkout@v2 | ||
- name: Set up Python ${{ matrix.python-version }} | ||
uses: actions/setup-python@v2 | ||
with: | ||
python-version: ${{ matrix.python-version }} | ||
- name: Install dependencies | ||
working-directory: ./embedding-calculator/ | ||
run: | | ||
sudo apt-get update && sudo apt-get install -y libjpeg-dev libpng-dev \ | ||
libtiff-dev libavformat-dev libpq-dev libfreeimage3 | ||
python -m pip install --upgrade pip | ||
python -m pip --no-cache-dir install -r requirements.txt | ||
python -m src.services.facescan.plugins.setup | ||
- name: Test with pytest | ||
working-directory: ./embedding-calculator/ | ||
run: | | ||
pip install pytest | ||
pip install pytest-cov | ||
python -m pytest -m "not performance" ./src |
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
registry=exadel/ | ||
postgres_username=postgres | ||
postgres_password=postgres | ||
postgres_db=frs | ||
postgres_domain=compreface-postgres-db | ||
postgres_port=5432 | ||
email_host=smtp.gmail.com | ||
email_username= | ||
email_from= | ||
email_password= | ||
enable_email_server=false | ||
save_images_to_db=true | ||
compreface_api_java_options=-Xmx8g | ||
compreface_admin_java_options=-Xmx8g | ||
ADMIN_VERSION=0.5.0 | ||
API_VERSION=0.5.0 | ||
FE_VERSION=0.5.0 | ||
CORE_VERSION=0.5.0-facenet |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
version: '3.4' | ||
|
||
volumes: | ||
postgres-data: | ||
|
||
services: | ||
compreface-postgres-db: | ||
image: postgres:11.5 | ||
container_name: "compreface-postgres-db" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_DB=${postgres_db} | ||
volumes: | ||
- postgres-data:/var/lib/postgresql/data | ||
|
||
compreface-admin: | ||
image: ${registry}compreface-admin:${ADMIN_VERSION} | ||
container_name: "compreface-admin" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- ENABLE_EMAIL_SERVER=${enable_email_server} | ||
- EMAIL_HOST=${email_host} | ||
- EMAIL_USERNAME=${email_username} | ||
- EMAIL_FROM=${email_from} | ||
- EMAIL_PASSWORD=${email_password} | ||
- ADMIN_JAVA_OPTS=${compreface_admin_java_options} | ||
depends_on: | ||
- compreface-postgres-db | ||
- compreface-api | ||
|
||
compreface-api: | ||
image: ${registry}compreface-api:${API_VERSION} | ||
container_name: "compreface-api" | ||
depends_on: | ||
- compreface-postgres-db | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- API_JAVA_OPTS=${compreface_api_java_options} | ||
- SAVE_IMAGES_TO_DB=${save_images_to_db} | ||
|
||
compreface-fe: | ||
image: ${registry}compreface-fe:${FE_VERSION} | ||
container_name: "compreface-ui" | ||
ports: | ||
- "8000:80" | ||
depends_on: | ||
- compreface-api | ||
- compreface-admin | ||
|
||
compreface-core: | ||
image: ${registry}compreface-core:${CORE_VERSION} | ||
container_name: "compreface-core" | ||
environment: | ||
- ML_PORT=3000 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
registry=exadel/ | ||
postgres_username=postgres | ||
postgres_password=postgres | ||
postgres_db=frs | ||
postgres_domain=compreface-postgres-db | ||
postgres_port=5432 | ||
email_host=smtp.gmail.com | ||
email_username= | ||
email_from= | ||
email_password= | ||
enable_email_server=false | ||
save_images_to_db=true | ||
compreface_api_java_options=-Xmx8g | ||
compreface_admin_java_options=-Xmx8g | ||
ADMIN_VERSION=0.5.0 | ||
API_VERSION=0.5.0 | ||
FE_VERSION=0.5.0 | ||
CORE_VERSION=0.5.0-mobilenet-gpu |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
version: '3.4' | ||
|
||
volumes: | ||
postgres-data: | ||
|
||
services: | ||
compreface-postgres-db: | ||
image: postgres:11.5 | ||
container_name: "compreface-postgres-db" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_DB=${postgres_db} | ||
volumes: | ||
- postgres-data:/var/lib/postgresql/data | ||
|
||
compreface-admin: | ||
image: ${registry}compreface-admin:${ADMIN_VERSION} | ||
container_name: "compreface-admin" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- ENABLE_EMAIL_SERVER=${enable_email_server} | ||
- EMAIL_HOST=${email_host} | ||
- EMAIL_USERNAME=${email_username} | ||
- EMAIL_FROM=${email_from} | ||
- EMAIL_PASSWORD=${email_password} | ||
- ADMIN_JAVA_OPTS=${compreface_admin_java_options} | ||
depends_on: | ||
- compreface-postgres-db | ||
- compreface-api | ||
|
||
compreface-api: | ||
image: ${registry}compreface-api:${API_VERSION} | ||
container_name: "compreface-api" | ||
depends_on: | ||
- compreface-postgres-db | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- API_JAVA_OPTS=${compreface_api_java_options} | ||
- SAVE_IMAGES_TO_DB=${save_images_to_db} | ||
|
||
compreface-fe: | ||
image: ${registry}compreface-fe:${FE_VERSION} | ||
container_name: "compreface-ui" | ||
ports: | ||
- "8000:80" | ||
depends_on: | ||
- compreface-api | ||
- compreface-admin | ||
|
||
compreface-core: | ||
image: ${registry}compreface-core:${CORE_VERSION} | ||
container_name: "compreface-core" | ||
runtime: nvidia | ||
environment: | ||
- ML_PORT=3000 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
registry=exadel/ | ||
postgres_username=postgres | ||
postgres_password=postgres | ||
postgres_db=frs | ||
postgres_domain=compreface-postgres-db | ||
postgres_port=5432 | ||
email_host=smtp.gmail.com | ||
email_username= | ||
email_from= | ||
email_password= | ||
enable_email_server=false | ||
save_images_to_db=true | ||
compreface_api_java_options=-Xmx8g | ||
compreface_admin_java_options=-Xmx8g | ||
ADMIN_VERSION=0.5.0 | ||
API_VERSION=0.5.0 | ||
FE_VERSION=0.5.0 | ||
CORE_VERSION=0.5.0-mobilenet |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
version: '3.4' | ||
|
||
volumes: | ||
postgres-data: | ||
|
||
services: | ||
compreface-postgres-db: | ||
image: postgres:11.5 | ||
container_name: "compreface-postgres-db" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_DB=${postgres_db} | ||
volumes: | ||
- postgres-data:/var/lib/postgresql/data | ||
|
||
compreface-admin: | ||
image: ${registry}compreface-admin:${ADMIN_VERSION} | ||
container_name: "compreface-admin" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- ENABLE_EMAIL_SERVER=${enable_email_server} | ||
- EMAIL_HOST=${email_host} | ||
- EMAIL_USERNAME=${email_username} | ||
- EMAIL_FROM=${email_from} | ||
- EMAIL_PASSWORD=${email_password} | ||
- ADMIN_JAVA_OPTS=${compreface_admin_java_options} | ||
depends_on: | ||
- compreface-postgres-db | ||
- compreface-api | ||
|
||
compreface-api: | ||
image: ${registry}compreface-api:${API_VERSION} | ||
container_name: "compreface-api" | ||
depends_on: | ||
- compreface-postgres-db | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- API_JAVA_OPTS=${compreface_api_java_options} | ||
- SAVE_IMAGES_TO_DB=${save_images_to_db} | ||
|
||
compreface-fe: | ||
image: ${registry}compreface-fe:${FE_VERSION} | ||
container_name: "compreface-ui" | ||
ports: | ||
- "8000:80" | ||
depends_on: | ||
- compreface-api | ||
- compreface-admin | ||
|
||
compreface-core: | ||
image: ${registry}compreface-core:${CORE_VERSION} | ||
container_name: "compreface-core" | ||
environment: | ||
- ML_PORT=3000 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
# List of custom-builds | ||
|
||
| Custom-build | Base library | CPU | GPU | Face detection model / accuracy on [WIDER Face (Hard)](https://paperswithcode.com/sota/face-detection-on-wider-face-hard) | Face recognition model / accuracy on [LFW](https://paperswithcode.com/sota/face-verification-on-labeled-faces-in-the) | Age and gender detection | Comment | | ||
| ------------------------------ | --------------------------------------------------------- | -------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ---------------------------------------------- | | ||
| FaceNet (default) | [FaceNet](https://github.com/davidsandberg/facenet) | x86 (AVX instructions) | not supported | MTCNN / 80.9% | FaceNet (20180402-114759) / 99.63% | Custom, the model is taken [here](https://github.com/GilLevi/AgeGenderDeepLearning) | For general purposes. Support CPU without AVX2 | | ||
| Mobilenet | [InsightFace](https://github.com/deepinsight/insightface) | x86 (AVX2 instructions) | not supported | RetinaFace-MobileNet0.25 / 82.5% | MobileFaceNet,ArcFace / 99.50% | InsightFace | The fastest model among CPU only models | | ||
| Mobilenet-gpu | [InsightFace](https://github.com/deepinsight/insightface) | x86 (AVX2 instructions) | GPU (CUDA required) | RetinaFace-MobileNet0.25 / 82.5% | MobileFaceNet,ArcFace / 99.50% | InsightFace | The fastest model | | ||
| SubCenter-ArcFace-r100 | [InsightFace](https://github.com/deepinsight/insightface) | x86 (AVX2 instructions) | not supported | retinaface_r50_v1 / 91.4% | arcface-r100-msfdrop75 / 99.80% | InsightFace | The most accurate model, but the most slow | | ||
| SubCenter-ArcFace-r100-gpu | [InsightFace](https://github.com/deepinsight/insightface) | x86 (AVX2 instructions) | GPU (CUDA required) | retinaface_r50_v1 / 91.4% | arcface-r100-msfdrop75 / 99.80% | InsightFace | The most accurate model | |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
registry=exadel/ | ||
postgres_username=postgres | ||
postgres_password=postgres | ||
postgres_db=frs | ||
postgres_domain=compreface-postgres-db | ||
postgres_port=5432 | ||
email_host=smtp.gmail.com | ||
email_username= | ||
email_from= | ||
email_password= | ||
enable_email_server=false | ||
save_images_to_db=true | ||
compreface_api_java_options=-Xmx8g | ||
compreface_admin_java_options=-Xmx8g | ||
ADMIN_VERSION=0.5.0 | ||
API_VERSION=0.5.0 | ||
FE_VERSION=0.5.0 | ||
CORE_VERSION=0.5.0-arcface-r100-gpu |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
version: '3.4' | ||
|
||
volumes: | ||
postgres-data: | ||
|
||
services: | ||
compreface-postgres-db: | ||
image: postgres:11.5 | ||
container_name: "compreface-postgres-db" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_DB=${postgres_db} | ||
volumes: | ||
- postgres-data:/var/lib/postgresql/data | ||
|
||
compreface-admin: | ||
image: ${registry}compreface-admin:${ADMIN_VERSION} | ||
container_name: "compreface-admin" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- ENABLE_EMAIL_SERVER=${enable_email_server} | ||
- EMAIL_HOST=${email_host} | ||
- EMAIL_USERNAME=${email_username} | ||
- EMAIL_FROM=${email_from} | ||
- EMAIL_PASSWORD=${email_password} | ||
- ADMIN_JAVA_OPTS=${compreface_admin_java_options} | ||
depends_on: | ||
- compreface-postgres-db | ||
- compreface-api | ||
|
||
compreface-api: | ||
image: ${registry}compreface-api:${API_VERSION} | ||
container_name: "compreface-api" | ||
depends_on: | ||
- compreface-postgres-db | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- API_JAVA_OPTS=${compreface_api_java_options} | ||
- SAVE_IMAGES_TO_DB=${save_images_to_db} | ||
|
||
compreface-fe: | ||
image: ${registry}compreface-fe:${FE_VERSION} | ||
container_name: "compreface-ui" | ||
ports: | ||
- "8000:80" | ||
depends_on: | ||
- compreface-api | ||
- compreface-admin | ||
|
||
compreface-core: | ||
image: ${registry}compreface-core:${CORE_VERSION} | ||
container_name: "compreface-core" | ||
runtime: nvidia | ||
environment: | ||
- ML_PORT=3000 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
registry=exadel/ | ||
postgres_username=postgres | ||
postgres_password=postgres | ||
postgres_db=frs | ||
postgres_domain=compreface-postgres-db | ||
postgres_port=5432 | ||
email_host=smtp.gmail.com | ||
email_username= | ||
email_from= | ||
email_password= | ||
enable_email_server=false | ||
save_images_to_db=true | ||
compreface_api_java_options=-Xmx8g | ||
compreface_admin_java_options=-Xmx8g | ||
ADMIN_VERSION=0.5.0 | ||
API_VERSION=0.5.0 | ||
FE_VERSION=0.5.0 | ||
CORE_VERSION=0.5.0-arcface-r100 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
version: '3.4' | ||
|
||
volumes: | ||
postgres-data: | ||
|
||
services: | ||
compreface-postgres-db: | ||
image: postgres:11.5 | ||
container_name: "compreface-postgres-db" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_DB=${postgres_db} | ||
volumes: | ||
- postgres-data:/var/lib/postgresql/data | ||
|
||
compreface-admin: | ||
image: ${registry}compreface-admin:${ADMIN_VERSION} | ||
container_name: "compreface-admin" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- ENABLE_EMAIL_SERVER=${enable_email_server} | ||
- EMAIL_HOST=${email_host} | ||
- EMAIL_USERNAME=${email_username} | ||
- EMAIL_FROM=${email_from} | ||
- EMAIL_PASSWORD=${email_password} | ||
- ADMIN_JAVA_OPTS=${compreface_admin_java_options} | ||
depends_on: | ||
- compreface-postgres-db | ||
- compreface-api | ||
|
||
compreface-api: | ||
image: ${registry}compreface-api:${API_VERSION} | ||
container_name: "compreface-api" | ||
depends_on: | ||
- compreface-postgres-db | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- API_JAVA_OPTS=${compreface_api_java_options} | ||
- SAVE_IMAGES_TO_DB=${save_images_to_db} | ||
|
||
compreface-fe: | ||
image: ${registry}compreface-fe:${FE_VERSION} | ||
container_name: "compreface-ui" | ||
ports: | ||
- "8000:80" | ||
depends_on: | ||
- compreface-api | ||
- compreface-admin | ||
|
||
compreface-core: | ||
image: ${registry}compreface-core:${CORE_VERSION} | ||
container_name: "compreface-core" | ||
environment: | ||
- ML_PORT=3000 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# CompreFace | ||
|
||
#### Getting started for Contributors: | ||
|
||
1. Install Docker and Docker-Compose | ||
2. Clone repository | ||
3. Open dev folder | ||
4. Run command: `sh start.sh` | ||
|
||
#### Tips for Windows (use Git Bash terminal) | ||
|
||
1. Turn of the git autocrlf with command: `git config --global core.autocrlf false` | ||
2. Make sure all your containers are down: `docker ps` | ||
3. In case some containers are working, they should be stopped: `docker-compose down` | ||
4. Clean all local datebases and images: `docker system prune --volumes` | ||
5. Go to Dev folder `cd dev` | ||
6. Run `sh start.sh` and make sure http://localhost:4200/ starts | ||
|
||
#### How to start for UI development: | ||
|
||
Open a new terminal window and run next commands: | ||
|
||
- Run `cd ui` | ||
- Run `npm install` (only for first time run) | ||
- Run `npm start` | ||
|
||
Go to http://localhost:4200/ and you can sign up with any credentials |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
version: '3.4' | ||
|
||
volumes: | ||
postgres-data: | ||
|
||
services: | ||
compreface-postgres-db: | ||
image: postgres:11.5 | ||
container_name: "compreface-postgres-db" | ||
ports: | ||
- "6432:5432" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_DB=${postgres_db} | ||
volumes: | ||
- postgres-data:/var/lib/postgresql/data | ||
|
||
compreface-admin: | ||
image: ${registry}compreface-admin:${ADMIN_VERSION} | ||
build: | ||
context: ../java | ||
dockerfile: ../dev/Dockerfile | ||
target: frs_crud | ||
container_name: "compreface-admin" | ||
ports: | ||
- "8081:8080" | ||
- "5006:5005" | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- ENABLE_EMAIL_SERVER=${enable_email_server} | ||
- EMAIL_HOST=${email_host} | ||
- EMAIL_USERNAME=${email_username} | ||
- EMAIL_FROM=${email_from} | ||
- EMAIL_PASSWORD=${email_password} | ||
- ADMIN_JAVA_OPTS=${compreface_admin_java_options} | ||
depends_on: | ||
- compreface-postgres-db | ||
- compreface-api | ||
|
||
compreface-api: | ||
image: ${registry}compreface-api:${API_VERSION} | ||
build: | ||
context: ../java | ||
dockerfile: ../dev/Dockerfile | ||
target: frs_core | ||
args: | ||
- ND4J_CLASSIFIER=${ND4J_CLASSIFIER} | ||
container_name: "compreface-api" | ||
ports: | ||
- "8082:8080" | ||
- "5005:5005" | ||
depends_on: | ||
- compreface-postgres-db | ||
environment: | ||
- POSTGRES_USER=${postgres_username} | ||
- POSTGRES_PASSWORD=${postgres_password} | ||
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db} | ||
- SPRING_PROFILES_ACTIVE=dev | ||
- API_JAVA_OPTS=${compreface_api_java_options} | ||
- SAVE_IMAGES_TO_DB=${save_images_to_db} | ||
|
||
compreface-fe: | ||
image: ${registry}compreface-fe:${FE_VERSION} | ||
build: | ||
context: ../ui | ||
dockerfile: docker-prod/Dockerfile | ||
container_name: "compreface-ui" | ||
ports: | ||
- "8000:80" | ||
depends_on: | ||
- compreface-api | ||
- compreface-admin | ||
|
||
compreface-core: | ||
image: ${registry}compreface-core:${CORE_VERSION} | ||
container_name: "compreface-core" | ||
ports: | ||
- "3300:3000" | ||
runtime: nvidia | ||
build: | ||
context: ../embedding-calculator | ||
args: | ||
- FACE_DETECTION_PLUGIN=insightface.FaceDetector@retinaface_r50_v1 | ||
- CALCULATION_PLUGIN=insightface.Calculator@arcface_r100_v1 | ||
- EXTRA_PLUGINS=insightface.LandmarksDetector,insightface.GenderDetector,insightface.AgeDetector | ||
- BASE_IMAGE=compreface-core-base:base-cuda100-py37 | ||
- GPU_IDX=0 | ||
environment: | ||
- ML_PORT=3000 |
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,3 @@ | ||
git checkout develop | ||
git pull | ||
#sed -i "s|%COMMIT_HASH%|$(git rev-parse --short HEAD)|g" /ui/src/app/features/footer/footer.component.html | ||
docker-compose -f docker-compose.yml up --build --scale compreface-postgres-db=0 | ||
#sed -i "s|%COMMIT_HASH%|$(git rev-parse --short HEAD)|g" ui/src/app/features/footer/footer.component.html | ||
docker-compose -f docker-compose.yml -f docker-compose.dev.ui.yml up --build & | ||
( cd ../ui && npm run start ) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
# Architecture and Scalability | ||
|
||
By default, CompreFace is delivered as docker-compose file, so you can easily start it with one command. However, CompreFace could be scaled up to distribute computations on different servers and achieve high availability. | ||
This section describes the architecture of CompreFace, each of its components, and suggestions on how to scale the system. | ||
|
||
## CompreFace architecture diagram | ||
|
||
![CompreFace architecture diagram](https://user-images.githubusercontent.com/3736126/107855144-5db83580-6e29-11eb-993a-46cdc0c82812.png) | ||
|
||
## Balancer + UI | ||
Container name in docker-compose file: compreface-fe | ||
|
||
This container runs Nginx that serves CompreFace UI. | ||
|
||
In the default config, it’s also used as the main gateway - Nginx proxies user requests to admin and api servers. | ||
|
||
## Admin server | ||
Container name in docker-compose file: compreface-admin | ||
|
||
Admin server is a Spring Boot application and it’s responsible for all operations that are done on UI. Admin server connects to PostgreSQL database to store the data. | ||
|
||
## Api servers | ||
Container name in docker-compose file: compreface-api | ||
|
||
Api servers handle all user API calls: face recognition, face detection, and face verification. | ||
|
||
It provides API key validation, proxies images to Embedding servers, and classifies the face. For face classification, we use the ND4J library. | ||
|
||
In the default config number of API servers is 1, but for production environments to increase possible bandwidth and to achieve high availability, there should be at least two such servers and they should be on different machines. The data synchronization is implemented via postgreSQL notifications, so if for example, you add a new face to a collection, all other servers will know about it and will be able to recognize this new face. | ||
|
||
Classification is not a very heavy operation as embedding calculation and in most cases doesn’t require GPU. API server connects to PostgreSQL database to store the data. | ||
|
||
## Embedding Servers | ||
Container name in docker-compose file: compreface-core | ||
|
||
Embedding server is responsible for running neural networks. It calculates embeddings from faces and makes all plugin recognitions like age and gender detection. These servers are stateless, they don't have a connection to a database and they don't require any synchronization between each other. | ||
|
||
In the default config number of API servers is 1, but for production environments to increase possible bandwidth and to achieve high availability, there should be at least two such servers and they should be on different machines. | ||
|
||
Running neural networks is a very heavy operation. The total performance of the system highly depends on these nodes. This is why we recommend using highly performant nodes to run Embedding Servers, ideally with GPU support. To learn more about how to run CompreFace with GPU, see custom builds documentation. | ||
|
||
## PostgreSQL | ||
Default docker-compose configuration includes postgreSQL database. | ||
|
||
If you want CompreFace to connect to your database, you need to provide such environment variables for compreface-admin and compreface-api containers: | ||
* POSTGRES_PASSWORD | ||
* POSTGRES_URL | ||
* POSTGRES_USER |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
# Configuration | ||
|
||
In the [release](https://github.com/exadel-inc/CompreFace/releases) archive and all custom builds there is a `.env` file with | ||
configuration options for CompreFace. For production systems | ||
we recommend looking through them and set up CompreFace accordingly | ||
|
||
* `registry` - this is the docker hub registry. For release and pre-build images, it should be set to `exadel/` value | ||
* `postgres_password` - password for Postgres database. It should be changed for production systems from the default value. | ||
* `postgres_domain` - the domain where Postgres database is run | ||
* `postgres_port` - Postgres database port | ||
* `enable_email_server` - if true, it will enable email verification for users. You should set email_host, email_username, and email_password variables for the correct work. | ||
* `email_host` - a host of the email provider. It should be set if `enable_email_server` variable is true | ||
* `email_username` - a username of email provider for authentication. It should be set if `enable_email_server` variable is true | ||
* `email_password` - a password of email provider for authentication. It should be set if `enable_email_server` variable is true | ||
* `email_from` - this value will see users in `from` fields when they receive emails from CompreFace. Corresponds to `From` field in rfc2822. Optional, if not set, then `email_username` will be used instead | ||
* `save_images_to_db` - should the CompreFace save photos to the database. Be careful, [migrations](Face-data-migration.md) could be run only if this value is `true` | ||
* `compreface_api_java_options` - java options of compreface-api container | ||
* `compreface_admin_java_options` - java options of compreface-admin container | ||
* `ADMIN_VERSION` - docker image tag of compreface-admin container | ||
* `API_VERSION` - docker image tag of compreface-api container | ||
* `FE_VERSION` - docker image tag of compreface-fe container | ||
* `CORE_VERSION` - docker image tag of compreface-core container |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
# Custom Builds | ||
|
||
There is always a trade off between the face recognition accuracy, max throughput of the system and even hardware support. | ||
|
||
By default, CompreFace release contains configuration that could be run on the widest variety of hardware. | ||
|
||
The downside of this build is that it's not optimized for the latest generations of CPU and doesn't support GPU. | ||
|
||
With custom-builds we aim to cover as many cases as we can. They are not tested as good as the default build, and we encourage community | ||
to report any bugs related to these builds. | ||
|
||
## List of custom-builds | ||
|
||
You can find the list of custom-builds [here](../custom-builds/README.md) | ||
|
||
|
||
## Contribution | ||
We also encourage community to share their own builds, we will add them to our list with notice that this is community build. | ||
|
||
## How to choose a build | ||
|
||
Different builds are fit for different purposes - some of them have higher accuracy, but the performance on CPU is low, others optimized | ||
for low-performance hardware and have acceptable accuracy. You have to make your own choice in this trade off. But generally you can | ||
follow this rules: | ||
|
||
- If you want to run real-time face recognition, we recommend choosing builds with GPU support. | ||
- If you need to run face recognition on old or low performance systems, we recommend to use builds with models originally created for | ||
mobile | ||
- Do not take blindly the most accurate model. The accuracy is not so different between models, but the required hardware resources | ||
could differ dramatically | ||
|
||
## How to run custom-builds | ||
|
||
Running custom-build is very similar to running the default build - all you need to do is to open the corresponding folder and run | ||
`docker-compose up -d`. | ||
|
||
Things to consider: | ||
- If you run CompreFace from the custom-build folder, it will create a new docker volume, | ||
so you won't see your saved information. To run custom-build with your previously saved information, | ||
you need to copy files from custom-build to folder with the original build (and replace the original files) | ||
- In most cases, face recognition models are not interchangeable, | ||
this means that all you saved examples from the old build won't work on new builds. | ||
See [migrations documentation](Face-data-migration.md) to know what is the options. | ||
- Do not run two instances of CompreFace simultaneously without changing the port. | ||
To change the port go to `docker-compose` file and change the post for `compreface-fe` container. | ||
|
||
|
||
## How to build your own custom-build | ||
|
||
### Custom models | ||
|
||
CompreFace supports two face recognition libraries - FaceNet and InsightFace. It means CompreFace can run any model that can run this | ||
libraries. All you need to do is | ||
1. Upload your model to Google Drive and add it to one the following files into the `Calculator` class: | ||
- /embedding-calculator/src/services/facescan/plugins/facenet/facenet.py | ||
- /embedding-calculator/src/services/facescan/plugins/insightface/insightface.py | ||
2. Take the `docker-compose` file from `/dev` folder as a template | ||
3. Specify new model name in build arguments. For more information look at [this documentation](https://github. | ||
com/exadel-inc/CompreFace/tree/master/embedding-calculator#run-service). E.g. here is a part of `docker-compose` file for building with custom model with GPU support: | ||
``` | ||
compreface-core: | ||
image: ${registry}compreface-core:${CORE_VERSION} | ||
container_name: "compreface-core" | ||
ports: | ||
- "3300:3000" | ||
runtime: nvidia | ||
build: | ||
context: ../embedding-calculator | ||
args: | ||
- FACE_DETECTION_PLUGIN=insightface.FaceDetector@retinaface_r50_v1 | ||
- CALCULATION_PLUGIN=insightface.Calculator@arcface_r100_v1 | ||
- EXTRA_PLUGINS=insightface.LandmarksDetector,insightface.GenderDetector,insightface.AgeDetector | ||
- BASE_IMAGE=compreface-core-base:base-cuda100-py37 | ||
- GPU_IDX=0 | ||
environment: | ||
- ML_PORT=3000 | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# Face Recognition Similarity Threshold | ||
|
||
The result of CompreFace face recognition and face verification services is a similarity between faces. Even if you upload faces of two different people, you will still receive the result, but the similarity will be low. The user must determine for himself whether this is the same person or not using similarity. | ||
The level of similarity the user accepts, as big enough, we call similarity threshold. | ||
|
||
## How to choose the face similarity threshold | ||
|
||
No Face Recognition Service has 100% accuracy, so there always will be errors in recognition. | ||
If a user chooses too low threshold, then some unknown faces will be recognized as known. | ||
If a user chooses too high threshold, then some known faces will be recognized as unknown. | ||
CompreFace calculates similarity in a way, so most correct guesses will be with a threshold of more than 0.5, and the most incorrect guesses will be with a threshold of less than 0.5. Still, we recommend for high-security systems set the threshold more than 0.5. | ||
This is the distribution of similarities for a custom dataset of 50,000 faces for FaceNet model (blue - is incorrect guesses, red is correct): | ||
<img src="https://user-images.githubusercontent.com/3736126/111870491-bb422380-898d-11eb-901d-0fad65eee69c.png" alt="distribution of similarities" width=800px style="padding: 10px;"> |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
# Face data migration | ||
|
||
## When do you need to migrate data | ||
|
||
When you upload a new known image to the Face Collection, | ||
CompreFace uses a neural network model to calculate an embedding (also known as face features), | ||
which is basically an array of 512 or 128 numbers. | ||
Then CompreFace saves it to the database to use in the future comparison - when you upload a face to recognize, | ||
CompreFace again calculates an embedding and compares it to saved embeddings. | ||
|
||
The important thing here is that every neural network model will calculate different embeddings. | ||
It means that it makes sense to compare embeddings calculated by the same neural network model. | ||
|
||
CompreFace doesn't change the neural network model during its work, so normally you don’t need to migrate your face data. | ||
If you want to try [custom build](Custom-builds.md), be very careful - look at the table [here](../custom-builds/README.md), | ||
column `Face recognition model` - if the model changed, you need to run a migration. | ||
|
||
## Limitations | ||
|
||
If you run CompreFace in the [“not saving images to database” mode](Configuration.md)(`save_images_to_db=false`), | ||
you won’t be able to migrate data as the original images are required for migration. | ||
|
||
The only solution here is to delete all images from Face Collection and upload them again. | ||
|
||
## How to perform a migration | ||
|
||
Current migration was written for internal usage and wasn’t tested enough, so please do a backup copy of the database and perform migration at your own risk. | ||
|
||
REST request to start migration: | ||
|
||
```shell | ||
curl -i -X POST \ | ||
'http://localhost:8000/api/v1/migrate' | ||
``` | ||
|
||
This rest endpoint is asynchronous, it will start the migration and return a response immediately. | ||
To understand if the migration is successful, please look at logs for “Migration successfully finished” text. |