-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
test: test preview sites for broken links (#1007)
* test: test preview sites for broken links * fix: ignore gifs too * test: add test for `_redirects` file * ci: implement preliminary workflow * fix: rename folder to `lambda` * ci: make script executable * chore: indicate code review together Co-authored-by: Debbie O'Brien <[email protected]> Co-authored-by: Debbie O'Brien <[email protected]>
- Loading branch information
1 parent
10aef6f
commit 03d24e4
Showing
12 changed files
with
510 additions
and
27 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| #!/bin/bash | ||
|
|
||
| STATE=$(echo $1 | tr '[:upper:]' '[:lower:]') | ||
| CONTEXT=$2 | ||
| DESCRIPTION=$3 | ||
| GITHUB_URL="https://github.com" | ||
| GITHUB_API_URL="https://api.github.com" | ||
|
|
||
| echo "Updating status to ${STATE} for ${CONTEXT} with description ${DESCRIPTION}." | ||
|
|
||
| curl --silent --show-error --fail \ | ||
| --trace ./${CONTEXT}.log \ | ||
| -X POST "${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/statuses/${GITHUB_SHA}" \ | ||
| -H "Authorization: token ${GITHUB_TOKEN}" \ | ||
| -H "Content-Type: text/json; charset=utf-8" \ | ||
| -d @- <<EOF | ||
| { | ||
| "state": "${STATE}", | ||
| "context": "${CONTEXT}", | ||
| "description": "${DESCRIPTION}" | ||
| } | ||
| EOF | ||
|
|
||
| sleep 5 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| name: Deployment tests | ||
| on: | ||
| repository_dispatch: | ||
| types: [check_links] | ||
|
|
||
| jobs: | ||
| check-links: | ||
| runs-on: ubuntu-18.04 | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
|
|
||
| - uses: actions/setup-node@v2-beta | ||
| with: | ||
| node-version: "14" | ||
|
|
||
| - uses: actions/cache@v2 | ||
| id: cache | ||
| with: | ||
| path: node_modules | ||
| key: ${{ hashFiles('yarn.lock') }} | ||
| restore-keys: ${{ runner.os }}-yarn | ||
|
|
||
| - name: Install dependencies | ||
| if: steps.cache.outputs.cache-hit != 'true' | ||
| run: yarn install --frozen-lockfile | ||
|
|
||
| - name: Crawl site | ||
| run: node -r esm scripts/crawl.js | ||
| env: | ||
| BASE_URL: ${{ github.event.client_payload.deploy_url }} | ||
|
|
||
| - name: Set status | ||
| if: always() | ||
| run: ./.github/bin/update-status.sh ${{ job.status }} check-links 'Finished checking links' | ||
| env: | ||
| GITHUB_TOKEN: ${{ github.token }} | ||
| GITHUB_SHA: ${{ github.event.client_payload.commit_ref }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| import fetch from 'node-fetch' | ||
| import jwt from 'jsonwebtoken' | ||
|
|
||
| exports.handler = async function ({ body, headers }) { | ||
| const signature = headers['X-Webhook-Signature'] | ||
|
|
||
| try { | ||
| jwt.verify(signature, process.env.SECRET_TOKEN || '') | ||
| } catch { | ||
| return { | ||
| statusCode: 403 | ||
| } | ||
| } | ||
|
|
||
| await fetch('https://api.github.com/repos/nuxt/nuxtjs.org/dispatches', { | ||
| method: 'post', | ||
| body: JSON.stringify({ | ||
| event_type: 'check_links', | ||
| client_payload: { | ||
| deploy_url: body.deploy_url, | ||
| commit_ref: body.commit_ref | ||
| } | ||
| }) | ||
| }) | ||
|
|
||
| return { | ||
| statusCode: 204 | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| import path from 'path' | ||
|
|
||
| import consola from 'consola' | ||
| import Crawler from 'crawler' | ||
| import fs from 'fs-extra' | ||
|
|
||
| const logger = consola.withTag('redirect-tester') | ||
|
|
||
| const redirects = fs | ||
| .readFileSync(path.resolve(__dirname, '../_redirects')) | ||
| .toString() | ||
| .split('\n') | ||
| .filter(redirect => redirect && !redirect.startsWith('#')) | ||
| .filter(redirect => redirect.startsWith('/')) | ||
| .map(redirect => redirect.split(' ')[1]) | ||
| .filter(redirect => redirect.startsWith('/')) | ||
| .map(redirect => 'https://nuxtjs.org' + redirect) | ||
|
|
||
| const crawler = new Crawler({ | ||
| maxConnections: 100, | ||
| callback(error, res, done) { | ||
| const { uri } = res.options | ||
| const { statusCode } = res.request.response | ||
|
|
||
| if (error || ![200, 301, 302].includes(statusCode)) { | ||
| logger.error('Error crawling', uri, `(status ${statusCode})`) | ||
| return done() | ||
| } | ||
|
|
||
| logger.success(uri) | ||
| done() | ||
| } | ||
| }) | ||
|
|
||
| logger.log('') | ||
| logger.info(`Checking \`internal redirects\`.`) | ||
|
|
||
| redirects.forEach(redirect => crawler.queue(redirect)) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| import Crawler from 'crawler' | ||
| import consola from 'consola' | ||
|
|
||
| const logger = consola.withTag('crawler') | ||
|
|
||
| const excludedExtensions = process.env.EXCLUDE | ||
| ? process.env.EXCLUDE.split(',') | ||
| : ['svg', 'png', 'jpg', 'sketch', 'ico', 'gif'] | ||
| const crawlExternal = !!process.env.CRAWL_EXTERNAL || false | ||
|
|
||
| let baseURL = process.env.BASE_URL || 'https://nuxtjs.org' | ||
| if (baseURL.endsWith('/')) baseURL = baseURL.slice(0, -1) | ||
| const startingURL = baseURL + '/' | ||
|
|
||
| // GLOBALS | ||
| const urls = new Set([startingURL]) | ||
| const referrers = {} | ||
| const erroredUrls = [] | ||
| const externalUrls = new Set() | ||
|
|
||
| /** | ||
| * @type {Crawler} crawler | ||
| */ | ||
| // eslint-disable-next-line | ||
| let crawler | ||
|
|
||
| /** | ||
| * @param {string} path | ||
| * @param {string | undefined} referrer | ||
| */ | ||
| function queue(path, referrer) { | ||
| const { pathname, origin } = new URL(path, referrer) | ||
|
|
||
| const url = `${origin}${pathname}` | ||
| if (!url || urls.has(url) || !crawler) return | ||
|
|
||
| const extension = url.split('.').pop() | ||
| if (excludedExtensions.includes(extension)) return | ||
|
|
||
| let external = false | ||
| if (origin !== baseURL) { | ||
| external = true | ||
| externalUrls.add(url) | ||
| if (!crawlExternal) return | ||
| } | ||
|
|
||
| urls.add(url) | ||
|
|
||
| if (referrer) referrers[url] = [...(referrers[url] || []), referrer] | ||
|
|
||
| if (external) | ||
| return crawler.queue({ | ||
| uri: url, | ||
| method: 'GET', | ||
| rateLimit: 2000 | ||
| }) | ||
| crawler.queue(url) | ||
| } | ||
|
|
||
| crawler = new Crawler({ | ||
| maxConnections: 100, | ||
| callback(error, res, done) { | ||
| const { $ } = res | ||
| const { uri } = res.options | ||
| const { statusCode } = res.request.response | ||
|
|
||
| if (error || ![200, 301, 302].includes(statusCode)) { | ||
| logger.error('Error crawling', uri, `(status ${statusCode})`) | ||
| if (referrers[uri]) logger.info(`${uri} referred by`, referrers[uri]) | ||
| erroredUrls.push(uri) | ||
| return done() | ||
| } | ||
|
|
||
| if (!$) { | ||
| logger.error('Could not parse', uri) | ||
| return done() | ||
| } | ||
|
|
||
| if (uri.includes(baseURL)) { | ||
| $(`a:not([href*=mailto])`).each((_, el) => queue(el.attribs.href, uri)) | ||
| } | ||
|
|
||
| logger.success(uri) | ||
| logger.debug(uri, `[${crawler.queueSize} / ${urls.size}]`) | ||
| if (crawler.queueSize === 1) { | ||
| logger.log('') | ||
| logger.info(`Checked \`${urls.size}\` pages.`) | ||
| // Tasks to run at the end. | ||
| if (erroredUrls.length) | ||
| throw new Error( | ||
| `\n\nErrors found when crawling ${erroredUrls.join(', ')}.` | ||
| ) | ||
| } | ||
| done() | ||
| } | ||
| }) | ||
|
|
||
| logger.log('') | ||
| logger.info( | ||
| `Checking \`${baseURL}\`${crawlExternal ? ' and external links' : ''}.` | ||
| ) | ||
| logger.info(`Ignoring file extensions: \`${excludedExtensions.join(', ')}.\`\n`) | ||
|
|
||
| crawler.queue(startingURL) |
Oops, something went wrong.