Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Recursive mlocks created in PosixFS #11093

Open
Michuu opened this issue Mar 4, 2025 · 0 comments
Open

Recursive mlocks created in PosixFS #11093

Michuu opened this issue Mar 4, 2025 · 0 comments
Labels
Type:Bug

Comments

@Michuu
Copy link

Michuu commented Mar 4, 2025 •
edited
Loading

I've recently was able to spin up oCIS 7.0.1 (docker) with the new PosixFS. It's certainly an excellent addition overall. I tried to stress-test it by uploading thousands of files. While overall it handled them, it seems that creation of a new file in the parent directory (personal space) of the user triggers creation and deletion of .mlocks for ALL files, i.e. in this directory as well as in all child directories recursively. On the other hand, creation of a new folder in the parent directory only triggers creation and deletion of .mlocks in this directory itself.
I am not sure if this is a bug or whether this may belong better to the https://github.com/cs3org/reva repository, but it clearly is a very problematic behavior which leads oCIS to make the server busy for several minutes if you have thousands of files.

Steps to reproduce

Consider the following directory structure, with folders A and B containing files A1.txt and B1.txt, respectively.

Image

Now create a new text file in the parent directory.
EDIT: It's worth noting that the same thing happens if you create files with web interface, or directly in the filesystem.

Expected behavior

The creation of .mlocks in the same folder is probably expected. The creation of .mlock for ALL files seems excessive.

Actual behavior

In actuality, there is the following "avalanche" of locks created:

2025/03/04 08:47:27 /data/users/username/,CREATE,New file.txt.mlock
2025/03/04 08:47:27 /data/users/username/,CREATE,New file.txt
2025/03/04 08:47:27 /data/users/,CREATE,username.mlock
2025/03/04 08:47:27 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:27 /data/users/,DELETE,username.mlock
2025/03/04 08:47:27 /data/users/,CREATE,username.mlock
2025/03/04 08:47:27 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:27 /data/users/username/,"CLOSE_WRITE,CLOSE",New file.txt.mlock
2025/03/04 08:47:27 /data/users/username/,DELETE,New file.txt.mlock
2025-03-04T08:47:27Z ERR resource info is missing checksum service=thumbnails
2025/03/04 08:47:28 /data/users/username/,CREATE,New file.txt.mlock
2025/03/04 08:47:28 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:28 /data/users/,DELETE,username.mlock
2025/03/04 08:47:28 /data/users/,CREATE,username.mlock
2025/03/04 08:47:28 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:28 /data/users/,DELETE,username.mlock
2025/03/04 08:47:28 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,.Trash.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",.Trash.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,.Trash.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,CREATE,files.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,"CLOSE_WRITE,CLOSE",files.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,DELETE,files.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,CREATE,info.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,"CLOSE_WRITE,CLOSE",info.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,DELETE,info.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,001.txt.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",001.txt.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,001.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,A.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",A.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,A.mlock
2025/03/04 08:47:28 /data/users/username/A/,CREATE,A1.txt.mlock
2025/03/04 08:47:28 /data/users/username/A/,"CLOSE_WRITE,CLOSE",A1.txt.mlock
2025/03/04 08:47:28 /data/users/username/A/,DELETE,A1.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,B.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",B.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,B.mlock
2025/03/04 08:47:28 /data/users/username/B/,CREATE,B1.txt.mlock
2025/03/04 08:47:28 /data/users/username/B/,"CLOSE_WRITE,CLOSE",B1.txt.mlock
2025/03/04 08:47:28 /data/users/username/B/,DELETE,B1.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,B.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",B.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,B.mlock
2025/03/04 08:47:28 /data/users/,CREATE,username.mlock
2025/03/04 08:47:28 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:28 /data/users/,DELETE,username.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,A.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",A.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,A.mlock
2025/03/04 08:47:37 /data/users/username/,CREATE,New file.txt.mlock
2025/03/04 08:47:37 /data/users/,CREATE,username.mlock
2025/03/04 08:47:37 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:37 /data/users/username/,"CLOSE_WRITE,CLOSE",New file.txt.mlock
2025/03/04 08:47:37 /data/users/username/,DELETE,New file.txt.mlock

Now, if I create a New Folder in the parent directory, the behavior is more reasonable:

2025/03/04 08:50:21 /data/users/,DELETE,username.flock
2025/03/04 08:50:21 /data/users/username/,CREATE,New folder.mlock
2025/03/04 08:50:21 /data/users/username/,"CREATE,ISDIR",New folder
2025/03/04 08:50:21 /data/users/username/,"CLOSE_WRITE,CLOSE",New folder.mlock
2025/03/04 08:50:21 /data/users/username/,DELETE,New folder.mlock
2025/03/04 08:50:21 /data/users/username/,"CLOSE_WRITE,CLOSE",New folder.mlock
2025/03/04 08:50:21 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:50:22 /data/users/username/,CREATE,New folder.mlock
2025/03/04 08:50:22 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:50:22 /data/users/username/,"CLOSE_WRITE,CLOSE",New folder.mlock
2025/03/04 08:50:22 /data/users/username/,DELETE,New folder.mlock
2025/03/04 08:50:22 /data/users/username/,CREATE,New folder.mlock
2025/03/04 08:50:22 /data/users/username/,"CLOSE_WRITE,CLOSE",New folder.mlock
2025/03/04 08:50:22 /data/users/username/,DELETE,New folder.mlock

Similarly, if I create a file in the A dir, there are no locks in the B dir.
Overall, this behavior is a big problem if you have thousands of files.

Setup

I use the docker-compose.yml presented below.

services:
  ocis:
    image: owncloud/ocis:7.0.1
    user: "1000:1000"
    environment:
      # INSECURE: needed if oCIS / Traefik is using self generated certificates
      OCIS_INSECURE: "true"

      # OCIS_URL: the external domain / ip address of oCIS (with protocol, must always be https)
      OCIS_URL: "https://ocis.example.com"
      GATEWAY_GRPC_ADDR: "0.0.0.0:9142"
      MICRO_REGISTRY_ADDRESS: "127.0.0.1:9233"
      NATS_NATS_HOST: "0.0.0.0"
      NATS_NATS_PORT: 9233

      STORAGE_USERS_DRIVER: "posix"
      STORAGE_USERS_POSIX_ROOT: "/data"
      STORAGE_USERS_POSIX_WATCH_TYPE: "inotifywait"
      STORAGE_USERS_ID_CACHE_STORE: "nats-js-kv"
      STORAGE_USERS_ID_CACHE_STORE_NODES: "localhost:9233"
      STORAGE_USERS_POSIX_USE_SPACE_GROUPS: "true"
      # OCIS_LOG_LEVEL: error / info / ... / debug
      OCIS_LOG_LEVEL: "error"
      OCIS_LOG_COLOR: "true"
      OCIS_LOG_PRETTY: "true"
      PROXY_ENABLE_BASIC_AUTH: "false"

      # OIDC with authentik
      OCIS_OIDC_ISSUER: "https://auth.example.com/application/o/ocis/"
      PROXY_OIDC_ISSUER: "https://auth.example.com/application/o/ocis/"
      PROXY_USER_OIDC_CLAIM: "preferred_username"
      WEB_OIDC_CLIENT_ID: "[redacted]"
      PROXY_OIDC_REWRITE_WELLKNOWN: "true"
      PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD: "none"
      WEB_OIDC_SCOPE: "openid profile email groups"
      PROXY_ROLE_ASSIGNMENT_DRIVER: "oidc"
      PROXY_ROLE_ASSIGNMENT_OIDC_CLAIM: "ocisRoles"
      PROXY_AUTOPROVISION_ACCOUNTS: "true"
      PROXY_CSP_CONFIG_FILE_LOCATION: '/etc/ocis/csp.yaml'
      OCIS_EXCLUDE_RUN_SERVICES: "idp"
      GRAPH_ASSIGN_DEFAULT_USER_ROLE: "false"
      GRAPH_USERNAME_MATCH: "none"
      PROXY_USER_CS3_CLAIM: "username"
      # SMTP
      NOTIFICATIONS_SMTP_HOST: "smtp.example.com"
      NOTIFICATIONS_SMTP_PORT: "465"
      NOTIFICATIONS_SMTP_AUTHENTICATION: "plain"
      NOTIFICATIONS_SMTP_SENDER: "OCIS notifications <[email protected]>"
      NOTIFICATIONS_SMTP_USERNAME: "[email protected]"
      NOTIFICATIONS_SMTP_PASSWORD: "password"
      NOTIFICATIONS_SMTP_ENCRYPTION: "ssltls"
      OCIS_ADD_RUN_SERVICES: "notifications"
      # SEARCH with tika
      SEARCH_EXTRACTOR_TYPE: "tika"
      SEARCH_EXTRACTOR_TIKA_TIKA_URL: "http://tika:9998"
      FRONTEND_FULL_TEXT_SEARCH_ENABLED: "true"
      SEARCH_ENGINE_BLEVE_DATA_PATH: "/var/lib/ocis/search"
      SEARCH_ENGINE_TYPE: "bleve"

    volumes:
      # mount the ocis config file inside the container
      - "./etc-ocis:/etc/ocis"
      - "./ocis-data:/data"
      - "./var-lib:/var/lib/ocis"
    expose:
      - 9200
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}.rule=Host(`ocis.example.eu`)"
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}.entrypoints=websecure"
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}.tls.certresolver=tls-resolver"
      - "traefik.http.services.${COMPOSE_PROJECT_NAME}.loadbalancer.server.port=9200"
      - "traefik.http.services.${COMPOSE_PROJECT_NAME}.loadbalancer.server.scheme=https"
      # cors
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}.middlewares=ocis-cors"
      - "traefik.http.middlewares.ocis-cors.headers.accesscontrolallowmethods=GET,OPTIONS,PUT"
      - "traefik.http.middlewares.ocis-cors.headers.accesscontrolalloworiginlist=https://ocis.example.com,https://auth.example.com"
      - "traefik.http.middlewares.ocis-cors.headers.accesscontrolmaxage=100"
      - "traefik.http.middlewares.ocis-cors.headers.addvaryheader=true"
      # MKCOL, MOVE body removal, workaround https://github.com/owncloud/ocis/issues/10809
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.rule=Host(`ocis.example.eu`) && (Method(`MKCOL`) || Method(`MOVE`))"
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.entrypoints=websecure"
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.tls.certresolver=tls-resolver"
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.middlewares=ocis-mkcol-body"
      - "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.service=${COMPOSE_PROJECT_NAME}"
      - "traefik.http.middlewares.ocis-mkcol-body.buffering.maxRequestBodyBytes=1"

  tika:
    image: apache/tika:3.1.0.0-full
    # release notes: https://tika.apache.org
    restart: always
    logging:
      driver: local

Additional considerations

The presented log files come from the docker logs. A separate issue seems to be that I am not able to disable them. They are there if if I set log level to warn or error. Is this expected?
@Michuu Michuu added the Type:Bug label Mar 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type:Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Michuu