Skip to content

Commit

Permalink
Merge pull request exadel-inc#503 from exadel-inc/0.5.0
Browse files Browse the repository at this point in the history
0.5.0 to master
pospielov authored Apr 25, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
2 parents 4b233d3 + fa7ac74 commit 8aa4654
Showing 891 changed files with 13,993 additions and 93,495 deletions.
10 changes: 6 additions & 4 deletions .env
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
registry=exadel/
postgres_username=postgres
postgres_password=postgres
postgres_db=frs
postgres_domain=compreface-postgres-db
postgres_port=5432
email_host=smtp.gmail.com
@@ -10,7 +12,7 @@ enable_email_server=false
save_images_to_db=true
compreface_api_java_options=-Xmx8g
compreface_admin_java_options=-Xmx8g
ADMIN_VERSION=0.4.1
API_VERSION=0.4.1
FE_VERSION=0.4.1
CORE_VERSION=0.4.1
ADMIN_VERSION=0.5.0
API_VERSION=0.5.0
FE_VERSION=0.5.0
CORE_VERSION=0.5.0
17 changes: 8 additions & 9 deletions .github/workflows/Push-Python-to-Dockerhub.yml
Original file line number Diff line number Diff line change
@@ -4,23 +4,22 @@ on:
workflow_dispatch:
inputs:
version:
description: Version
default: v1.0.0
description: Version (e.g., 0.4.1)
required: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: docker login
- name: Build the Docker image and push
env:
VERSION: ${{ github.event.inputs.version }}
DOCKER_USER: ${{secrets.DOCKER_HUB_LOGIN}}
DOCKER_PASSWORD: ${{secrets.DOCKER_HUB_PWD}}
run: |
docker login -u "${DOCKER_HUB_LOGIN}" -p "${DOCKER_HUB_PWD}"
- name: Build the Docker image
DOCKER_REGISTRY: ${{secrets.DOCKER_HUB_LOGIN}}/
working-directory: ./embedding-calculator/
run: |
make build-images VERSION={{ github.event.inputs.version }}
- name: Docker Push
run: docker push exadel/compreface-core
make build-images
docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}"
docker push --all-tags ${DOCKER_REGISTRY}compreface-core
2 changes: 1 addition & 1 deletion .github/workflows/cla.yml
Original file line number Diff line number Diff line change
@@ -18,6 +18,6 @@ jobs:
PERSONAL_ACCESS_TOKEN : ${{ secrets.PERSONAL_ACCESS_TOKEN }}
with:
path-to-signatures: 'signatures/cla.json'
path-to-cla-document: 'https://github.com/exadel-inc/CompreFace/blob/develop/CLA.md'
path-to-cla-document: 'https://github.com/exadel-inc/CompreFace/blob/master/CLA.md'
branch: 'CLA-signatures'
allowlist: bot*
36 changes: 36 additions & 0 deletions .github/workflows/unit-tests-on-nodejs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Unit tests on Node.js

on:
push:
branches: [ develop ]
pull_request:
branches: [ develop ]

jobs:
build:
runs-on: ubuntu-latest

strategy:
matrix:
node-version: [14.10]

steps:
- uses: actions/checkout@v2
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
- name: Cache Node.js modules
uses: actions/cache@v2
with:
path: ~/.npm
key: ${{ runner.OS }}-node-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.OS }}-node-
${{ runner.OS }}-
- name: Install dependencies
working-directory: ./ui
run: npm install
- name: Run tests
working-directory: ./ui
run: npm test -- --no-watch --no-progress --browsers=ChromeHeadlessCI
35 changes: 35 additions & 0 deletions .github/workflows/unit-tests-on-python.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Unit tests on Python test

on:
push:
branches: [ develop ]
pull_request:
branches: [ develop ]
jobs:
build:

runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.7]

steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
working-directory: ./embedding-calculator/
run: |
sudo apt-get update && sudo apt-get install -y libjpeg-dev libpng-dev \
libtiff-dev libavformat-dev libpq-dev libfreeimage3
python -m pip install --upgrade pip
python -m pip --no-cache-dir install -r requirements.txt
python -m src.services.facescan.plugins.setup
- name: Test with pytest
working-directory: ./embedding-calculator/
run: |
pip install pytest
pip install pytest-cov
python -m pytest -m "not performance" ./src
556 changes: 60 additions & 496 deletions README.md

Large diffs are not rendered by default.

This file was deleted.

42 changes: 0 additions & 42 deletions admin/src/main/java/com/exadel/frs/helpers/AppRoleConverter.java

This file was deleted.

This file was deleted.

This file was deleted.

173 changes: 0 additions & 173 deletions admin/src/main/java/com/exadel/frs/service/ModelService.java

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

12 changes: 0 additions & 12 deletions api/src/test/resources/application.yml

This file was deleted.

18 changes: 18 additions & 0 deletions custom-builds/FaceNet/.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
registry=exadel/
postgres_username=postgres
postgres_password=postgres
postgres_db=frs
postgres_domain=compreface-postgres-db
postgres_port=5432
email_host=smtp.gmail.com
email_username=
email_from=
email_password=
enable_email_server=false
save_images_to_db=true
compreface_api_java_options=-Xmx8g
compreface_admin_java_options=-Xmx8g
ADMIN_VERSION=0.5.0
API_VERSION=0.5.0
FE_VERSION=0.5.0
CORE_VERSION=0.5.0-facenet
61 changes: 61 additions & 0 deletions custom-builds/FaceNet/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
version: '3.4'

volumes:
postgres-data:

services:
compreface-postgres-db:
image: postgres:11.5
container_name: "compreface-postgres-db"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_DB=${postgres_db}
volumes:
- postgres-data:/var/lib/postgresql/data

compreface-admin:
image: ${registry}compreface-admin:${ADMIN_VERSION}
container_name: "compreface-admin"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- ENABLE_EMAIL_SERVER=${enable_email_server}
- EMAIL_HOST=${email_host}
- EMAIL_USERNAME=${email_username}
- EMAIL_FROM=${email_from}
- EMAIL_PASSWORD=${email_password}
- ADMIN_JAVA_OPTS=${compreface_admin_java_options}
depends_on:
- compreface-postgres-db
- compreface-api

compreface-api:
image: ${registry}compreface-api:${API_VERSION}
container_name: "compreface-api"
depends_on:
- compreface-postgres-db
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- API_JAVA_OPTS=${compreface_api_java_options}
- SAVE_IMAGES_TO_DB=${save_images_to_db}

compreface-fe:
image: ${registry}compreface-fe:${FE_VERSION}
container_name: "compreface-ui"
ports:
- "8000:80"
depends_on:
- compreface-api
- compreface-admin

compreface-core:
image: ${registry}compreface-core:${CORE_VERSION}
container_name: "compreface-core"
environment:
- ML_PORT=3000
18 changes: 18 additions & 0 deletions custom-builds/Mobilenet-gpu/.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
registry=exadel/
postgres_username=postgres
postgres_password=postgres
postgres_db=frs
postgres_domain=compreface-postgres-db
postgres_port=5432
email_host=smtp.gmail.com
email_username=
email_from=
email_password=
enable_email_server=false
save_images_to_db=true
compreface_api_java_options=-Xmx8g
compreface_admin_java_options=-Xmx8g
ADMIN_VERSION=0.5.0
API_VERSION=0.5.0
FE_VERSION=0.5.0
CORE_VERSION=0.5.0-mobilenet-gpu
62 changes: 62 additions & 0 deletions custom-builds/Mobilenet-gpu/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
version: '3.4'

volumes:
postgres-data:

services:
compreface-postgres-db:
image: postgres:11.5
container_name: "compreface-postgres-db"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_DB=${postgres_db}
volumes:
- postgres-data:/var/lib/postgresql/data

compreface-admin:
image: ${registry}compreface-admin:${ADMIN_VERSION}
container_name: "compreface-admin"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- ENABLE_EMAIL_SERVER=${enable_email_server}
- EMAIL_HOST=${email_host}
- EMAIL_USERNAME=${email_username}
- EMAIL_FROM=${email_from}
- EMAIL_PASSWORD=${email_password}
- ADMIN_JAVA_OPTS=${compreface_admin_java_options}
depends_on:
- compreface-postgres-db
- compreface-api

compreface-api:
image: ${registry}compreface-api:${API_VERSION}
container_name: "compreface-api"
depends_on:
- compreface-postgres-db
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- API_JAVA_OPTS=${compreface_api_java_options}
- SAVE_IMAGES_TO_DB=${save_images_to_db}

compreface-fe:
image: ${registry}compreface-fe:${FE_VERSION}
container_name: "compreface-ui"
ports:
- "8000:80"
depends_on:
- compreface-api
- compreface-admin

compreface-core:
image: ${registry}compreface-core:${CORE_VERSION}
container_name: "compreface-core"
runtime: nvidia
environment:
- ML_PORT=3000
18 changes: 18 additions & 0 deletions custom-builds/Mobilenet/.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
registry=exadel/
postgres_username=postgres
postgres_password=postgres
postgres_db=frs
postgres_domain=compreface-postgres-db
postgres_port=5432
email_host=smtp.gmail.com
email_username=
email_from=
email_password=
enable_email_server=false
save_images_to_db=true
compreface_api_java_options=-Xmx8g
compreface_admin_java_options=-Xmx8g
ADMIN_VERSION=0.5.0
API_VERSION=0.5.0
FE_VERSION=0.5.0
CORE_VERSION=0.5.0-mobilenet
61 changes: 61 additions & 0 deletions custom-builds/Mobilenet/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
version: '3.4'

volumes:
postgres-data:

services:
compreface-postgres-db:
image: postgres:11.5
container_name: "compreface-postgres-db"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_DB=${postgres_db}
volumes:
- postgres-data:/var/lib/postgresql/data

compreface-admin:
image: ${registry}compreface-admin:${ADMIN_VERSION}
container_name: "compreface-admin"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- ENABLE_EMAIL_SERVER=${enable_email_server}
- EMAIL_HOST=${email_host}
- EMAIL_USERNAME=${email_username}
- EMAIL_FROM=${email_from}
- EMAIL_PASSWORD=${email_password}
- ADMIN_JAVA_OPTS=${compreface_admin_java_options}
depends_on:
- compreface-postgres-db
- compreface-api

compreface-api:
image: ${registry}compreface-api:${API_VERSION}
container_name: "compreface-api"
depends_on:
- compreface-postgres-db
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- API_JAVA_OPTS=${compreface_api_java_options}
- SAVE_IMAGES_TO_DB=${save_images_to_db}

compreface-fe:
image: ${registry}compreface-fe:${FE_VERSION}
container_name: "compreface-ui"
ports:
- "8000:80"
depends_on:
- compreface-api
- compreface-admin

compreface-core:
image: ${registry}compreface-core:${CORE_VERSION}
container_name: "compreface-core"
environment:
- ML_PORT=3000
9 changes: 9 additions & 0 deletions custom-builds/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# List of custom-builds

| Custom-build | Base library | CPU | GPU | Face detection model / accuracy on [WIDER Face (Hard)](https://paperswithcode.com/sota/face-detection-on-wider-face-hard) | Face recognition model / accuracy on [LFW](https://paperswithcode.com/sota/face-verification-on-labeled-faces-in-the) | Age and gender detection | Comment |
| ------------------------------ | --------------------------------------------------------- | -------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ---------------------------------------------- |
| FaceNet (default) | [FaceNet](https://github.com/davidsandberg/facenet) | x86 (AVX instructions) | not supported | MTCNN / 80.9% | FaceNet (20180402-114759) / 99.63% | Custom, the model is taken [here](https://github.com/GilLevi/AgeGenderDeepLearning) | For general purposes. Support CPU without AVX2 |
| Mobilenet | [InsightFace](https://github.com/deepinsight/insightface) | x86 (AVX2 instructions) | not supported | RetinaFace-MobileNet0.25 / 82.5% | MobileFaceNet,ArcFace / 99.50% | InsightFace | The fastest model among CPU only models |
| Mobilenet-gpu | [InsightFace](https://github.com/deepinsight/insightface) | x86 (AVX2 instructions) | GPU (CUDA required) | RetinaFace-MobileNet0.25 / 82.5% | MobileFaceNet,ArcFace / 99.50% | InsightFace | The fastest model |
| SubCenter-ArcFace-r100 | [InsightFace](https://github.com/deepinsight/insightface) | x86 (AVX2 instructions) | not supported | retinaface_r50_v1 / 91.4% | arcface-r100-msfdrop75 / 99.80% | InsightFace | The most accurate model, but the most slow |
| SubCenter-ArcFace-r100-gpu | [InsightFace](https://github.com/deepinsight/insightface) | x86 (AVX2 instructions) | GPU (CUDA required) | retinaface_r50_v1 / 91.4% | arcface-r100-msfdrop75 / 99.80% | InsightFace | The most accurate model |
18 changes: 18 additions & 0 deletions custom-builds/SubCenter-ArcFace-r100-gpu/.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
registry=exadel/
postgres_username=postgres
postgres_password=postgres
postgres_db=frs
postgres_domain=compreface-postgres-db
postgres_port=5432
email_host=smtp.gmail.com
email_username=
email_from=
email_password=
enable_email_server=false
save_images_to_db=true
compreface_api_java_options=-Xmx8g
compreface_admin_java_options=-Xmx8g
ADMIN_VERSION=0.5.0
API_VERSION=0.5.0
FE_VERSION=0.5.0
CORE_VERSION=0.5.0-arcface-r100-gpu
62 changes: 62 additions & 0 deletions custom-builds/SubCenter-ArcFace-r100-gpu/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
version: '3.4'

volumes:
postgres-data:

services:
compreface-postgres-db:
image: postgres:11.5
container_name: "compreface-postgres-db"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_DB=${postgres_db}
volumes:
- postgres-data:/var/lib/postgresql/data

compreface-admin:
image: ${registry}compreface-admin:${ADMIN_VERSION}
container_name: "compreface-admin"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- ENABLE_EMAIL_SERVER=${enable_email_server}
- EMAIL_HOST=${email_host}
- EMAIL_USERNAME=${email_username}
- EMAIL_FROM=${email_from}
- EMAIL_PASSWORD=${email_password}
- ADMIN_JAVA_OPTS=${compreface_admin_java_options}
depends_on:
- compreface-postgres-db
- compreface-api

compreface-api:
image: ${registry}compreface-api:${API_VERSION}
container_name: "compreface-api"
depends_on:
- compreface-postgres-db
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- API_JAVA_OPTS=${compreface_api_java_options}
- SAVE_IMAGES_TO_DB=${save_images_to_db}

compreface-fe:
image: ${registry}compreface-fe:${FE_VERSION}
container_name: "compreface-ui"
ports:
- "8000:80"
depends_on:
- compreface-api
- compreface-admin

compreface-core:
image: ${registry}compreface-core:${CORE_VERSION}
container_name: "compreface-core"
runtime: nvidia
environment:
- ML_PORT=3000
18 changes: 18 additions & 0 deletions custom-builds/SubCenter-ArcFace-r100/.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
registry=exadel/
postgres_username=postgres
postgres_password=postgres
postgres_db=frs
postgres_domain=compreface-postgres-db
postgres_port=5432
email_host=smtp.gmail.com
email_username=
email_from=
email_password=
enable_email_server=false
save_images_to_db=true
compreface_api_java_options=-Xmx8g
compreface_admin_java_options=-Xmx8g
ADMIN_VERSION=0.5.0
API_VERSION=0.5.0
FE_VERSION=0.5.0
CORE_VERSION=0.5.0-arcface-r100
61 changes: 61 additions & 0 deletions custom-builds/SubCenter-ArcFace-r100/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
version: '3.4'

volumes:
postgres-data:

services:
compreface-postgres-db:
image: postgres:11.5
container_name: "compreface-postgres-db"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_DB=${postgres_db}
volumes:
- postgres-data:/var/lib/postgresql/data

compreface-admin:
image: ${registry}compreface-admin:${ADMIN_VERSION}
container_name: "compreface-admin"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- ENABLE_EMAIL_SERVER=${enable_email_server}
- EMAIL_HOST=${email_host}
- EMAIL_USERNAME=${email_username}
- EMAIL_FROM=${email_from}
- EMAIL_PASSWORD=${email_password}
- ADMIN_JAVA_OPTS=${compreface_admin_java_options}
depends_on:
- compreface-postgres-db
- compreface-api

compreface-api:
image: ${registry}compreface-api:${API_VERSION}
container_name: "compreface-api"
depends_on:
- compreface-postgres-db
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- API_JAVA_OPTS=${compreface_api_java_options}
- SAVE_IMAGES_TO_DB=${save_images_to_db}

compreface-fe:
image: ${registry}compreface-fe:${FE_VERSION}
container_name: "compreface-ui"
ports:
- "8000:80"
depends_on:
- compreface-api
- compreface-admin

compreface-core:
image: ${registry}compreface-core:${CORE_VERSION}
container_name: "compreface-core"
environment:
- ML_PORT=3000
8 changes: 8 additions & 0 deletions dev/.env
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
registry=
postgres_password=postgres
postgres_username=postgres
postgres_db=frs
postgres_domain=compreface-postgres-db
postgres_port=5432
email_host=smtp.gmail.com
@@ -14,3 +16,9 @@ ADMIN_VERSION=latest
API_VERSION=latest
FE_VERSION=latest
CORE_VERSION=latest

# ND4J library classifier values:
# * linux-x86_64, windows-x86_64, macosx-x86_64 - old CPUs (pre 2012) and low power x86 (Atom, Celeron): no AVX support (usually)
# * linux-x86_64-avx2, windows-x86_64-avx2, macosx-x86_64-avx2 - most modern x86 CPUs: AVX2 is supported
# * linux-x86_64-avx512 - some high-end server CPUs: AVX512 may be supported
ND4J_CLASSIFIER=linux-x86_64
9 changes: 5 additions & 4 deletions dev/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
FROM maven:3.6.3-jdk-11-slim as build
ARG ND4J_CLASSIFIER
WORKDIR /workspace/compreface
LABEL intermidiate_frs=true
COPY pom.xml .
COPY api/pom.xml api/
COPY admin/pom.xml admin/
RUN mvn -B clean install -DskipTests -Dcheckstyle.skip -Dasciidoctor.skip -Djacoco.skip -Dmaven.gitcommitid.skip -Dspring-boot.repackage.skip -Dmaven.exec.skip=true -Dmaven.install.skip -Dmaven.resources.skip
COPY api api
COPY admin admin
RUN mvn package -Dmaven.test.skip=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true
COPY common common
RUN mvn -B clean package -Dmaven.test.skip=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true -Dnd4j.classifier=$ND4J_CLASSIFIER

FROM openjdk:11.0.8-jre-slim as frs_core
ARG DIR=/workspace/compreface
@@ -17,4 +16,6 @@ ENTRYPOINT ["sh","-c","java $API_JAVA_OPTS -agentlib:jdwp=transport=dt_socket,se
FROM openjdk:11.0.8-jre-slim as frs_crud
ARG DIR=/workspace/compreface
COPY --from=build ${DIR}/admin/target/*.jar /home/app.jar
ARG APPERY_API_KEY
ENV APPERY_API_KEY ${APPERY_API_KEY}
ENTRYPOINT ["sh","-c","java $ADMIN_JAVA_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar /home/app.jar"]
27 changes: 27 additions & 0 deletions dev/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# CompreFace

#### Getting started for Contributors:

1. Install Docker and Docker-Compose
2. Clone repository
3. Open dev folder
4. Run command: `sh start.sh`

#### Tips for Windows (use Git Bash terminal)

1. Turn of the git autocrlf with command: `git config --global core.autocrlf false`
2. Make sure all your containers are down: `docker ps`
3. In case some containers are working, they should be stopped: `docker-compose down`
4. Clean all local datebases and images: `docker system prune --volumes`
5. Go to Dev folder `cd dev`
6. Run `sh start.sh` and make sure http://localhost:4200/ starts

#### How to start for UI development:

Open a new terminal window and run next commands:

- Run `cd ui`
- Run `npm install` (only for first time run)
- Run `npm start`

Go to http://localhost:4200/ and you can sign up with any credentials
93 changes: 93 additions & 0 deletions dev/docker-compose-gpu.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
version: '3.4'

volumes:
postgres-data:

services:
compreface-postgres-db:
image: postgres:11.5
container_name: "compreface-postgres-db"
ports:
- "6432:5432"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_DB=${postgres_db}
volumes:
- postgres-data:/var/lib/postgresql/data

compreface-admin:
image: ${registry}compreface-admin:${ADMIN_VERSION}
build:
context: ../java
dockerfile: ../dev/Dockerfile
target: frs_crud
container_name: "compreface-admin"
ports:
- "8081:8080"
- "5006:5005"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- ENABLE_EMAIL_SERVER=${enable_email_server}
- EMAIL_HOST=${email_host}
- EMAIL_USERNAME=${email_username}
- EMAIL_FROM=${email_from}
- EMAIL_PASSWORD=${email_password}
- ADMIN_JAVA_OPTS=${compreface_admin_java_options}
depends_on:
- compreface-postgres-db
- compreface-api

compreface-api:
image: ${registry}compreface-api:${API_VERSION}
build:
context: ../java
dockerfile: ../dev/Dockerfile
target: frs_core
args:
- ND4J_CLASSIFIER=${ND4J_CLASSIFIER}
container_name: "compreface-api"
ports:
- "8082:8080"
- "5005:5005"
depends_on:
- compreface-postgres-db
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- API_JAVA_OPTS=${compreface_api_java_options}
- SAVE_IMAGES_TO_DB=${save_images_to_db}

compreface-fe:
image: ${registry}compreface-fe:${FE_VERSION}
build:
context: ../ui
dockerfile: docker-prod/Dockerfile
container_name: "compreface-ui"
ports:
- "8000:80"
depends_on:
- compreface-api
- compreface-admin

compreface-core:
image: ${registry}compreface-core:${CORE_VERSION}
container_name: "compreface-core"
ports:
- "3300:3000"
runtime: nvidia
build:
context: ../embedding-calculator
args:
- FACE_DETECTION_PLUGIN=insightface.FaceDetector@retinaface_r50_v1
- CALCULATION_PLUGIN=insightface.Calculator@arcface_r100_v1
- EXTRA_PLUGINS=insightface.LandmarksDetector,insightface.GenderDetector,insightface.AgeDetector
- BASE_IMAGE=compreface-core-base:base-cuda100-py37
- GPU_IDX=0
environment:
- ML_PORT=3000
22 changes: 12 additions & 10 deletions dev/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -10,25 +10,26 @@ services:
ports:
- "6432:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_DB=frs
- POSTGRES_DB=${postgres_db}
volumes:
- postgres-data:/var/lib/postgresql/data

compreface-admin:
image: ${registry}compreface-admin:${ADMIN_VERSION}
build:
context: ..
dockerfile: dev/Dockerfile
context: ../java
dockerfile: ../dev/Dockerfile
target: frs_crud
container_name: "compreface-admin"
ports:
- "8081:8080"
- "5006:5005"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/frs
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- ENABLE_EMAIL_SERVER=${enable_email_server}
- EMAIL_HOST=${email_host}
@@ -43,18 +44,21 @@ services:
compreface-api:
image: ${registry}compreface-api:${API_VERSION}
build:
context: ..
dockerfile: dev/Dockerfile
context: ../java
dockerfile: ../dev/Dockerfile
target: frs_core
args:
- ND4J_CLASSIFIER=${ND4J_CLASSIFIER}
container_name: "compreface-api"
ports:
- "8082:8080"
- "5005:5005"
depends_on:
- compreface-postgres-db
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/frs
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- API_JAVA_OPTS=${compreface_api_java_options}
- SAVE_IMAGES_TO_DB=${save_images_to_db}
@@ -64,8 +68,6 @@ services:
build:
context: ../ui
dockerfile: docker-prod/Dockerfile
volumes:
- ${compreface-fe-volume}
container_name: "compreface-ui"
ports:
- "8000:80"
3 changes: 0 additions & 3 deletions dev/start--dev.sh

This file was deleted.

7 changes: 3 additions & 4 deletions dev/start.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
git checkout develop
git pull
#sed -i "s|%COMMIT_HASH%|$(git rev-parse --short HEAD)|g" /ui/src/app/features/footer/footer.component.html
docker-compose -f docker-compose.yml up --build --scale compreface-postgres-db=0
#sed -i "s|%COMMIT_HASH%|$(git rev-parse --short HEAD)|g" ui/src/app/features/footer/footer.component.html
docker-compose -f docker-compose.yml -f docker-compose.dev.ui.yml up --build &
( cd ../ui && npm run start )
12 changes: 7 additions & 5 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -8,18 +8,19 @@ services:
image: postgres:11.5
container_name: "compreface-postgres-db"
environment:
- POSTGRES_USER=postgres
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_DB=frs
- POSTGRES_DB=${postgres_db}
volumes:
- postgres-data:/var/lib/postgresql/data

compreface-admin:
image: ${registry}compreface-admin:${ADMIN_VERSION}
container_name: "compreface-admin"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/frs
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- ENABLE_EMAIL_SERVER=${enable_email_server}
- EMAIL_HOST=${email_host}
@@ -37,8 +38,9 @@ services:
depends_on:
- compreface-postgres-db
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/frs
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- API_JAVA_OPTS=${compreface_api_java_options}
- SAVE_IMAGES_TO_DB=${save_images_to_db}
@@ -56,4 +58,4 @@ services:
image: ${registry}compreface-core:${CORE_VERSION}
container_name: "compreface-core"
environment:
- ML_PORT=3000
- ML_PORT=3000
48 changes: 48 additions & 0 deletions docs/Architecture-and-scalability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Architecture and Scalability

By default, CompreFace is delivered as docker-compose file, so you can easily start it with one command. However, CompreFace could be scaled up to distribute computations on different servers and achieve high availability.
This section describes the architecture of CompreFace, each of its components, and suggestions on how to scale the system.

## CompreFace architecture diagram

![CompreFace architecture diagram](https://user-images.githubusercontent.com/3736126/107855144-5db83580-6e29-11eb-993a-46cdc0c82812.png)

## Balancer + UI
Container name in docker-compose file: compreface-fe

This container runs Nginx that serves CompreFace UI.

In the default config, it’s also used as the main gateway - Nginx proxies user requests to admin and api servers.

## Admin server
Container name in docker-compose file: compreface-admin

Admin server is a Spring Boot application and it’s responsible for all operations that are done on UI. Admin server connects to PostgreSQL database to store the data.

## Api servers
Container name in docker-compose file: compreface-api

Api servers handle all user API calls: face recognition, face detection, and face verification.

It provides API key validation, proxies images to Embedding servers, and classifies the face. For face classification, we use the ND4J library.

In the default config number of API servers is 1, but for production environments to increase possible bandwidth and to achieve high availability, there should be at least two such servers and they should be on different machines. The data synchronization is implemented via postgreSQL notifications, so if for example, you add a new face to a collection, all other servers will know about it and will be able to recognize this new face.

Classification is not a very heavy operation as embedding calculation and in most cases doesn’t require GPU. API server connects to PostgreSQL database to store the data.

## Embedding Servers
Container name in docker-compose file: compreface-core

Embedding server is responsible for running neural networks. It calculates embeddings from faces and makes all plugin recognitions like age and gender detection. These servers are stateless, they don't have a connection to a database and they don't require any synchronization between each other.

In the default config number of API servers is 1, but for production environments to increase possible bandwidth and to achieve high availability, there should be at least two such servers and they should be on different machines.

Running neural networks is a very heavy operation. The total performance of the system highly depends on these nodes. This is why we recommend using highly performant nodes to run Embedding Servers, ideally with GPU support. To learn more about how to run CompreFace with GPU, see custom builds documentation.

## PostgreSQL
Default docker-compose configuration includes postgreSQL database.

If you want CompreFace to connect to your database, you need to provide such environment variables for compreface-admin and compreface-api containers:
* POSTGRES_PASSWORD
* POSTGRES_URL
* POSTGRES_USER
22 changes: 22 additions & 0 deletions docs/Configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Configuration

In the [release](https://github.com/exadel-inc/CompreFace/releases) archive and all custom builds there is a `.env` file with
configuration options for CompreFace. For production systems
we recommend looking through them and set up CompreFace accordingly

* `registry` - this is the docker hub registry. For release and pre-build images, it should be set to `exadel/` value
* `postgres_password` - password for Postgres database. It should be changed for production systems from the default value.
* `postgres_domain` - the domain where Postgres database is run
* `postgres_port` - Postgres database port
* `enable_email_server` - if true, it will enable email verification for users. You should set email_host, email_username, and email_password variables for the correct work.
* `email_host` - a host of the email provider. It should be set if `enable_email_server` variable is true
* `email_username` - a username of email provider for authentication. It should be set if `enable_email_server` variable is true
* `email_password` - a password of email provider for authentication. It should be set if `enable_email_server` variable is true
* `email_from` - this value will see users in `from` fields when they receive emails from CompreFace. Corresponds to `From` field in rfc2822. Optional, if not set, then `email_username` will be used instead
* `save_images_to_db` - should the CompreFace save photos to the database. Be careful, [migrations](Face-data-migration.md) could be run only if this value is `true`
* `compreface_api_java_options` - java options of compreface-api container
* `compreface_admin_java_options` - java options of compreface-admin container
* `ADMIN_VERSION` - docker image tag of compreface-admin container
* `API_VERSION` - docker image tag of compreface-api container
* `FE_VERSION` - docker image tag of compreface-fe container
* `CORE_VERSION` - docker image tag of compreface-core container
77 changes: 77 additions & 0 deletions docs/Custom-builds.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Custom Builds

There is always a trade off between the face recognition accuracy, max throughput of the system and even hardware support.

By default, CompreFace release contains configuration that could be run on the widest variety of hardware.

The downside of this build is that it's not optimized for the latest generations of CPU and doesn't support GPU.

With custom-builds we aim to cover as many cases as we can. They are not tested as good as the default build, and we encourage community
to report any bugs related to these builds.

## List of custom-builds

You can find the list of custom-builds [here](../custom-builds/README.md)


## Contribution
We also encourage community to share their own builds, we will add them to our list with notice that this is community build.

## How to choose a build

Different builds are fit for different purposes - some of them have higher accuracy, but the performance on CPU is low, others optimized
for low-performance hardware and have acceptable accuracy. You have to make your own choice in this trade off. But generally you can
follow this rules:

- If you want to run real-time face recognition, we recommend choosing builds with GPU support.
- If you need to run face recognition on old or low performance systems, we recommend to use builds with models originally created for
mobile
- Do not take blindly the most accurate model. The accuracy is not so different between models, but the required hardware resources
could differ dramatically

## How to run custom-builds

Running custom-build is very similar to running the default build - all you need to do is to open the corresponding folder and run
`docker-compose up -d`.

Things to consider:
- If you run CompreFace from the custom-build folder, it will create a new docker volume,
so you won't see your saved information. To run custom-build with your previously saved information,
you need to copy files from custom-build to folder with the original build (and replace the original files)
- In most cases, face recognition models are not interchangeable,
this means that all you saved examples from the old build won't work on new builds.
See [migrations documentation](Face-data-migration.md) to know what is the options.
- Do not run two instances of CompreFace simultaneously without changing the port.
To change the port go to `docker-compose` file and change the post for `compreface-fe` container.


## How to build your own custom-build

### Custom models

CompreFace supports two face recognition libraries - FaceNet and InsightFace. It means CompreFace can run any model that can run this
libraries. All you need to do is
1. Upload your model to Google Drive and add it to one the following files into the `Calculator` class:
- /embedding-calculator/src/services/facescan/plugins/facenet/facenet.py
- /embedding-calculator/src/services/facescan/plugins/insightface/insightface.py
2. Take the `docker-compose` file from `/dev` folder as a template
3. Specify new model name in build arguments. For more information look at [this documentation](https://github.
com/exadel-inc/CompreFace/tree/master/embedding-calculator#run-service). E.g. here is a part of `docker-compose` file for building with custom model with GPU support:
```
compreface-core:
image: ${registry}compreface-core:${CORE_VERSION}
container_name: "compreface-core"
ports:
- "3300:3000"
runtime: nvidia
build:
context: ../embedding-calculator
args:
- FACE_DETECTION_PLUGIN=insightface.FaceDetector@retinaface_r50_v1
- CALCULATION_PLUGIN=insightface.Calculator@arcface_r100_v1
- EXTRA_PLUGINS=insightface.LandmarksDetector,insightface.GenderDetector,insightface.AgeDetector
- BASE_IMAGE=compreface-core-base:base-cuda100-py37
- GPU_IDX=0
environment:
- ML_PORT=3000
```
13 changes: 13 additions & 0 deletions docs/Face-Recognition-Similarity-Threshold.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Face Recognition Similarity Threshold

The result of CompreFace face recognition and face verification services is a similarity between faces. Even if you upload faces of two different people, you will still receive the result, but the similarity will be low. The user must determine for himself whether this is the same person or not using similarity.
The level of similarity the user accepts, as big enough, we call similarity threshold.

## How to choose the face similarity threshold

No Face Recognition Service has 100% accuracy, so there always will be errors in recognition.
If a user chooses too low threshold, then some unknown faces will be recognized as known.
If a user chooses too high threshold, then some known faces will be recognized as unknown.
CompreFace calculates similarity in a way, so most correct guesses will be with a threshold of more than 0.5, and the most incorrect guesses will be with a threshold of less than 0.5. Still, we recommend for high-security systems set the threshold more than 0.5.
This is the distribution of similarities for a custom dataset of 50,000 faces for FaceNet model (blue - is incorrect guesses, red is correct):
<img src="https://user-images.githubusercontent.com/3736126/111870491-bb422380-898d-11eb-901d-0fad65eee69c.png" alt="distribution of similarities" width=800px style="padding: 10px;">
37 changes: 37 additions & 0 deletions docs/Face-data-migration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Face data migration

## When do you need to migrate data

When you upload a new known image to the Face Collection,
CompreFace uses a neural network model to calculate an embedding (also known as face features),
which is basically an array of 512 or 128 numbers.
Then CompreFace saves it to the database to use in the future comparison - when you upload a face to recognize,
CompreFace again calculates an embedding and compares it to saved embeddings.

The important thing here is that every neural network model will calculate different embeddings.
It means that it makes sense to compare embeddings calculated by the same neural network model.

CompreFace doesn't change the neural network model during its work, so normally you don’t need to migrate your face data.
If you want to try [custom build](Custom-builds.md), be very careful - look at the table [here](../custom-builds/README.md),
column `Face recognition model` - if the model changed, you need to run a migration.

## Limitations

If you run CompreFace in the [“not saving images to database” mode](Configuration.md)(`save_images_to_db=false`),
you won’t be able to migrate data as the original images are required for migration.

The only solution here is to delete all images from Face Collection and upload them again.

## How to perform a migration

Current migration was written for internal usage and wasn’t tested enough, so please do a backup copy of the database and perform migration at your own risk.

REST request to start migration:

```shell
curl -i -X POST \
'http://localhost:8000/api/v1/migrate'
```

This rest endpoint is asynchronous, it will start the migration and return a response immediately.
To understand if the migration is successful, please look at logs for “Migration successfully finished” text.
Loading

0 comments on commit 8aa4654

Please sign in to comment.