[Sveltekit] Quirrel job fails on production #1154
Comments
|
So I managed to find the piece of code related to error logging, it was in owl repo: pipeline.publish(event, `${_queueId}:${_jobId}:${errorString}`);
pipeline.publish(_queueId, `${event}:${_jobId}:${errorString}`);
pipeline.publish(`${_queueId}:${_jobId}`, `${event}:${errorString}`);
pipeline.publish(`${_queueId}:${_jobId}:${event}`, errorString);Now the question is where and when they are published? I don't see it on the logs or any relevant error message that can put me in the right direction on fixing the issue. |
|
Seems I figured out something from the logs, particularly the following: Quirrel is trying to send a curl -include -X POST "https://my-domain/quirrel" -H "Accept: application/json" -d "cedc:encoded-data"Received the following message: {"message":"Cross-site POST form submissions are forbidden"}Sveltekit has enabled csrf protection by default. This is Disabling the CSRF protection first came into mind although it is not recommended. Disabled it so I could test things out, ran the build and started simulating production locally but it is still not working. It seems sending a POST request from seeing the logs but in fact it might not be able to send anything at all? Sending the POST through curl request does trigger the job tho. The updated curl request is now like this: # This will only work for development since quirrel requires `X-Quirrel-Signature` header in production
curl -include -X POST "http://localhost:3000/quirrel" -H "Accept: application/json" -H "Origin: http://localhost:3000" -d "cedc:encoded-data"Now I need to figure out why quirrel is not sending the request... |
|
Hi @lnfel! Good work digging through the logs, it looks like you're on track to figure it out. As a workaround, disabling CSRF protection sounds good. This is where Quirrel makes the HTTP request, and it doesn't specify an quirrel/src/api/worker/index.ts Lines 124 to 129 in 72ffb07 So it looks like maybe we need to add it there. Do you want to open a PR for that? You should be able to get the value for |
|
Really appreciate the input @Skn0tt. I'm working on it now. Added the whole // src/api/worker/index.ts
let { tokenId, endpoint } = decodeQueueDescriptor(job.queue);
// omitted code...
const headers: Record<string, string> = {
"Content-Type": "text/plain",
"x-quirrel-meta": JSON.stringify({
id: job.id,
count: job.count,
exclusive: job.exclusive,
retry: job.retry,
nextRepetition: ack.nextExecutionDate,
}),
"Origin": endpoint, // The change
};Tried building Quirrel QUIRREL_TOKEN=aeda6e05-...
QUIRREL_BASE_URL=http://localhost:3000
QUIRREL_API_URL=http://localhost:9181Then fired off quirrel enqueue on my app. Same thing happened, it logged Any tips on how to test this locally? From what I read on the code the // dx-logger.ts
startingExecution(...): () => void { ... }
test(response: any): void { console.log(chalk`Response: ${response}`) } |
|
I'm not sure what's the best way to debug this. Maybe you could You mentioned that > new URL("https://test.com/test").origin
"https://test.com" |
|
I have written a sveltekit handle to allow CORS from post requests for // src/hooks.server.js
import { sequence } from '@sveltejs/kit/hooks'
async function svelteHandleCors({ event, resolve }) {
// Apply CORS header for specified route endpoints
console.log("API route: ", event.url.pathname.startsWith('/api'))
console.log("Quirrel route: ", event.url.pathname.startsWith('/quirrel'))
if (event.url.pathname.startsWith('/api') || event.url.pathname.startsWith('/quirrel')) {
// Required for CORS to work
console.log("Request method: ", event.request.method)
if (event.request.method === 'OPTIONS') {
return new Response(null, {
headers: {
'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE, PATCH, OPTIONS',
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Headers': '*',
}
})
}
}
const response = await resolve(event)
if (event.url.pathname.startsWith('/api') || event.url.pathname.startsWith('/quirrel')) {
response.headers.append('Access-Control-Allow-Origin', '*')
}
return response
}
export const handle = sequence(
svelteHandleCors,
)Did a lot of testing, so far here are the results:✅ Sveltekit running on vite development server at port 5173:
❌ Sveltekit built with the following env config running using node at port 3000:QUIRREL_TOKEN=aeda6e05-...
QUIRREL_BASE_URL=http://localhost:3000
QUIRREL_API_URL=http://localhost:9181Command used to run the node server after building the app: ORIGIN=http://localhost:3000 BODY_SIZE_LIMIT=0 node build
How do we know the request was never sent? I've added a logger after Promise.all . ❗Sveltekit built with the following env config running using vite preview at port 4173:QUIRREL_TOKEN=aeda6e05-...
QUIRREL_BASE_URL=http://localhost:4173
QUIRREL_API_URL=http://localhost:9181Command used to run vite preview after building the app: npx vite preview --open
From what I see in the code, there is this My conclusion is that Quirrel is working when vite server is used but it breaks when the app is run using node server. This is very tricky as there is no error or log that shows what is wrong. Running |
|
I also don't think CORS plays a role here, since CORS is only enforced by the user-agent. Quirrel doesn't enforce CORS. Out of interest, what happens when you disable CSRF protection in your Svelte App? Also, have you tried my recommendation from #1154 (comment)? What was the result? |
|
I did tried it: const headers: Record<string, string> = {
"Content-Type": "text/plain",
"x-quirrel-meta": JSON.stringify({
id: job.id,
count: job.count,
exclusive: job.exclusive,
retry: job.retry,
nextRepetition: ack.nextExecutionDate,
}),
"Origin": new URL(endpoint).origin,
};It shows on the logs too:
Yes I think so too, that is why I handled the CORS on the sveltekit app instead. Disabling CORS on the app allows POST requests without same ORIGIN. // svelte.config.js
import adapterNode from '@sveltejs/adapter-node'
import { vitePreprocess } from '@sveltejs/kit/vite'
const config = {
kit: {
adapter: adapterNode(),
csrf: {
checkOrigin: false // disable ORIGIN check to enable CORS
}
},
preprocess: vitePreprocess()
}
export default config;Here are the curl request I made: # Server
# ORIGIN=http://localhost:3000 BODY_SIZE_LIMIT=0 node build
# Also note that `process.env.NODE_ENV` is undefined and the results are the same if I set it to development
curl -include -X POST "localhost:3000/quirrel" \
-H "Content-Type: text/plain" \
-H "x-quirrel-meta: {\"id\":\"4e6c1dcc-a13f-410b-8016-4d9fad6232c8\",\"count\":1,\"exclusive\":false,\"retry\":[]}" \
-d "{\"artifactCollectionId\":\"artc_WUkZO6\",\"locals\":{}}"
HTTP/1.1 200 OK
access-control-allow-origin: *
content-type: text/plain;charset=UTF-8
Date: Fri, 07 Jul 2023 02:13:59 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked
OK
# Server
# NODE_ENV=production ORIGIN=http://localhost:3000 BODY_SIZE_LIMIT=0 node build
# X-Quirrel-Signature is required for production environment, Quirrel should have it sent but this is not the issue
curl -include -X POST "localhost:3000/quirrel" \
-H "Content-Type: text/plain" \
-H "x-quirrel-meta: {\"id\":\"4e6c1dcc-a13f-410b-8016-4d9fad6232c8\",\"count\":1,\"exclusive\":false,\"retry\":[]}" \
-d "{\"artifactCollectionId\":\"artc_0CFUsr\",\"locals\":{}}"
HTTP/1.1 401 Unauthorized
access-control-allow-origin: *
content-type: text/plain;charset=UTF-8
Date: Fri, 07 Jul 2023 02:19:33 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked
Signature missingOnly one thing I can't get my head around with is when I run the app using node, the quirrel/src/api/worker/index.ts Lines 123 to 131 in 154e1a1 Issue might be unrelated to Quirrel but with how sveltekit builds the endpoint during build time (but curl works, so quirrel fetch should also work). |
|
I'm having trouble following. For me to help, please share a minimal reproduction repository, with steps on how to reproduce the behaviour you're seeing and what you'd expect to happen instead. |
|
@Skn0tt I have good news, while every effort I did on simulating production environment on my local machine was hopeless, I did managed to make the production instance of sveltekit app work with Quirrel hosted on railway. I should have tested this on production sooner 😅. I am sure setting origin check fixed the issue on production: // svelte.config.js
import adapterNode from '@sveltejs/adapter-node'
import { vitePreprocess } from '@sveltejs/kit/vite'
const config = {
kit: {
adapter: adapterNode(),
csrf: {
checkOrigin: false // disable ORIGIN check to enable CORS
}
},
preprocess: vitePreprocess()
}
export default config;Thank you so much for guiding me! |
Bug Report
Quirrel works well on development environment but fails on production (job never fires). The quirrel instance is hosted on railway following the guide on https://docs.quirrel.dev/deployment/railway and checking the deploy logs gave me the following:
I am not really sure what the log is trying to say but it seemed quirrel does received the request but was not able to run the job?
This is how I use quirrel:
The code above is just a simple job that adds new data to database, not sure why it does not work in production.
Quirrel env variables are set like this inside the project's .env file and also set this as shared variables on the railway quirrel instance:
Environment
Additional notes: Also tried to simulate production environment locally using localhost:9181 as quirrel instance and app at localhost:3000, same thing also happens (job never fires) but I can't find logs of quirrel so it was really hard to debug locally.
The text was updated successfully, but these errors were encountered: