Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Self-Hosting Quirrel enqueue returns 500 error on AWS while trying to enqueue. #1030

Closed
RyomaDKishimoto opened this issue Sep 13, 2022 · 1 comment

Comments

@RyomaDKishimoto
Copy link

RyomaDKishimoto commented Sep 13, 2022

Bug Report

Current Behavior

Self-Hosting Quirrel enqueue works fine locally. but on the AWS returns 500 error and CloudWatch shows error logs below.

  • Process to error result
{
    "level": 30,
    "time": 1663039586809,
    "pid": 8,
    "hostname": "ip-10-0-3-184.ap-northeast-1.compute.internal",
    "reqId": "req-3o6",
    "req": {
        "method": "POST",
        "url": "/queues/http%3A%2F%2Fapp.test.dx.local%3A3000%2Fapi%2Fqueues%2FcreateAllPdfQueue",
        "hostname": "worker.test.dx.local:9181",
        "remoteAddress": "10.0.2.144",
        "remotePort": 44416
    },
    "msg": "incoming request"
}

{
    "level": 20,
    "time": 1663039586810,
    "pid": 8,
    "hostname": "ip-10-0-3-184.ap-northeast-1.compute.internal",
    "module": "owl",
    "job": {
        "queue": "NAME_OF_TOKEN;http%3A%2F%2Fapp.test.dx.local%3A3000%2Fapi%2Fqueues%2FcreateAllPdfQueue",
        "id": "c4595ba9-5524-48c8-802b-f1246269d941",
        "payload": "0bab:RpQ1K2VTA1rwrVu9/2PmTw==:ojyZPVX2veW+zHnnWrAzALge4LG1BX2wmnc=:hLir494SFlnPM0Bj4McyFA==",
        "retry": [
            60000,
            300000,
            600000
        ]
    },
    "msg": "Producer: Enqueueing"
}

{
    "level": 30,
    "time": 1663039586816,
    "pid": 8,
    "hostname": "ip-10-0-3-184.ap-northeast-1.compute.internal",
    "reqId": "req-3o6",
    "res": {
        "statusCode": 500  🤔
    },
    "responseTime": 7.614914998412132,
    "msg": "request completed"
}

  • Error result.
Unexpected response while trying to enqueue

{
    "body": "0bab:+SGPIgNCFRuNXtbiIs7COA==:Tx7msPnesf9kj+J39/Lm2cBoqU3X4eUB8W8=:hxtuHYl58DKaH4ZrrOW/Vg==",
    "retry": [
        60000,
        300000,
        600000
    ]
}
" to http://app.test.dx.local:3000: 
{
    "error": 500,
    "message": "Internal Server Error",
    "sentry": "eeffcc6c03024371a6de829b59c666cb"
}

Input Code

  • docker-compose.yml
version: '3'
services:
  db:
    image: mysql:8
    cap_add:
      - SYS_NICE
    restart: unless-stopped
    environment:
      MYSQL_USER: ***
      MYSQL_PASSWORD: ***
      MYSQL_DATABASE: ***
      MYSQL_ROOT_PASSWORD: ***
    volumes:
      - ./tmp/db:/var/lib/mysql
      - ./docker-compose/mysql/master/conf.d:/etc/mysql/conf.d
      - ./docker-compose/mysql/master/init_data:/docker-entrypoint-initdb.d
    ports:
      - '3306:3306'
  slave:
    image: mysql:8
    cap_add:
      - SYS_NICE
    restart: unless-stopped
    environment:
      SOURCE_MYSQL_HOST: ***
      SOURCE_MYSQL_USER: ***
      SOURCE_MYSQL_PASSWORD: ***
      MYSQL_ROOT_PASSWORD: ***
    volumes:
      - ./tmp/db-slave:/var/lib/mysql
      - ./docker-compose/mysql/slave/conf.d:/etc/mysql/conf.d
      - ./docker-compose/mysql/slave/init_data:/docker-entrypoint-initdb.d
    ports:
      - '3308:3306'
  smtp:
    image: schickling/mailcatcher
    ports:
      - '1080:1080'
      - '1025:1025'
  app:
    build: .
    command: bash -c "yarn && yarn migrate deploy && yarn blitz db seed && yarn run quirrel ci && yarn dev -p 3000"
    volumes:
      - .:/dx
    ports:
      - '3000:3000'
    environment:
      - CHOKIDAR_USEPOLLING=true
      - DATABASE_URL=***
      - DATABASE_URL_RO=***
      - MAIL_HOST=smtp
      - QUIRREL_API_URL=http://quirrel:9181
      - QUIRREL_BASE_URL=http://app:3000
      - S3_ENDPOINT_DEV=***
      - AWS_ACCESS_KEY_ID=***
      - AWS_SECRET_ACCESS_KEY=***
    depends_on:
      - db
      - slave
      - storage
      - quirrel
  quirrel:
    image: ghcr.io/quirrel-dev/quirrel:1.9.0
    environment:
      PASSPHRASES: ***
      REDIS_URL: redis://redis
    ports:
      - '9181:9181'
    depends_on:
      - redis
  redis:
    image: 'redis:latest'
    ports:
      - '6379:6379'
    volumes:
      - './tmp/redis:/data'
  • env
QUIRREL_API_URL=http://localhost:9181
QUIRREL_BASE_URL=http://localhost:3000  
QUIRREL_ENCRYPTION_SECRET=***   (setting on AWS side as well)
# QUIRREL_TOKEN=     (setting on AWS side as well)
  • a.ts
await createPdfFileQueue.enqueue(
    {
      fileId: file.id,
      userId: user.id,
    },
    {
      retry: ['1min', '5min', '10min'],
    },
  );
  • b.ts
const createAllPdfQueue = Queue(
  'api/queues/createAllPdfQueue',
  async (
    { fileId, userId }: { fileId: number; userId: number },
    { count, retry },
  ) => {
    const file = await writeClient.file.update({
      where: { id: file.id },
      data: { pdfStatus: 'HOGE' },
    });
    try {
      const pdfs = await createPdf(file);
    } catch (e) {
      console.error(e);
      throw e;
    }
  },
);
export default createAllPdfQueue;
  • app/manifest.yml
name: app
type: Load Balanced Web Service

# Distribute traffic to your service.
http:
  # Requests to this path will be forwarded to your service.
  # To match all requests you can use the "/" path.
  path: '/'
  healthcheck:
    path: '/'
    interval: 60s
    timeout: 30s
    unhealthy_threshold: 5
    grace_period: 300s
  allowed_source_ips: []


image:
  # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#image-build
  build:
    dockerfile: Dockerfile
  # Port exposed through your container to route traffic to it.
  port: 3000

cpu: 256 # Number of CPU units for the task.
memory: 1024 # Amount of memory in MiB used by the task.
count: 1 # Number of tasks that should be running in your service.
exec: true # Enable running commands in your container.

network:
  vpc:
    placement: private

secrets:
  DATABASE_PASSWORD:
    secretsmanager: '${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/DatabasePassword:password::'
  SESSION_SECRET_KEY:
    secretsmanager: '${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/SessionSecretKey:sessionSecretKey::'
  QUIRREL_TOKEN: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/api/QUIRREL_TOKEN # 二度目の起動のときにサーバーから取得してSystemManagerに設定しておく
  QUIRREL_ENCRYPTION_SECRET:
    secretsmanager: '${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/QuirrelEncryptionSecret:quirrelEncryptionSecret::'
  API_TOKEN: /copilot/${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/secrets/api/API_TOKEN

variables:
  NODE_ENV: production
  QUIRREL_API_URL: http://worker.${COPILOT_ENVIRONMENT_NAME}.${COPILOT_APPLICATION_NAME}.local:9181
  QUIRREL_BASE_URL: http://app.${COPILOT_ENVIRONMENT_NAME}.${COPILOT_APPLICATION_NAME}.local:3000

environments:
  prod:
    http:
      alias: dx.com
    variables:
      CLIENT_ORIGIN: https://dx.com
      MAIL_FROM_SUPPORT: [email protected]
  test:
    http:
      alias: test.dx.com
    variables:
      CLIENT_ORIGIN: https://test.dx.com
      MAIL_FROM_SUPPORT: [email protected]

  • woker/manifest.yml
name: worker
type: Backend Service

# Your service does not allow any traffic.

# Configuration for your containers and service.
image:
  location: ghcr.io/quirrel-dev/quirrel:sha-5d02551 # Quirrel:The Task Queueing Solution for Serverless.
  port: 9181
  healthcheck:
    command: ['CMD-SHELL', 'curl -f http://localhost:9181 || exit 1']
    interval: 10s
    retries: 1
    timeout: 5s
    start_period: 100s

cpu: 256 # Number of CPU units for the task.
memory: 512 # Amount of memory in MiB used by the task.
count: 1 # Number of tasks that should be running in your service.
exec: true # Enable running commands in your container.

network:
  vpc:
    placement: private
# Optional fields for more advanced use-cases.
#
#variables:                    # Pass environment variables as key value pairs.
#  LOG_LEVEL: info

secrets: # Pass secrets from AWS Systems Manager (SSM) Parameter Store.
  PASSPHRASES:
    secretsmanager: '${COPILOT_APPLICATION_NAME}/${COPILOT_ENVIRONMENT_NAME}/Passphrases:passphrases::'

  • woker/addons/quirrel-secrets.yml

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.

Resources:
  QuirrelEncryptionSecret:
    Type: 'AWS::SecretsManager::Secret'
    Properties:
      Description: Quirrel Encryption secret created by AWS CloudFormation.
      Name: !Sub ${App}/${Env}/QuirrelEncryptionSecret
      GenerateSecretString:
        SecretStringTemplate: '{}'
        GenerateStringKey: quirrelEncryptionSecret
        PasswordLength: 32
        ExcludePunctuation: Yes
        ExcludeUppercase: Yes

  QuirrelPassphrasesSecret:
    Type: 'AWS::SecretsManager::Secret'
    Properties:
      Description: Quirrel Passphrases secret created by AWS CloudFormation.
      Name: !Sub ${App}/${Env}/Passphrases
      GenerateSecretString:
        SecretStringTemplate: '{}'
        GenerateStringKey: passphrases
        PasswordLength: 32
        ExcludePunctuation: Yes
        ExcludeUppercase: Yes

Expected behavior/code

  1. Quirrel in a.ts triggered
  2. run code inside b.ts and call const file = await readClient.file.findFirst

Environment

  • Quirrel version:1.9.0(already tried, 1.9.3 and sha-5d02551)
  • Blitz.js
  • AWS
  • AWS copilot
  • Docker
@lnfel
Copy link

lnfel commented Jun 26, 2023

How did you solved it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants