This is the backend API for a Discord Server Search App with improved functiionality such as searching by multiple tags and by exlusion tags.
The python script found in /scripts/scrape.py
is adapted from: https://github.com/DiscordFederation/DisboardScraper
client
|---src // minimalist react client for demo of backend purposes only
scripts
|---scrape.py // scraper called in node.js
|---scrape_all_servers.py // script to populate mongoDB
server
|---app.js // load app
|---app.log // logging for server
|---connect.js // connect to mongoDB
|---helpers.js // parsing, formatting, and other helpers
|---server.js // server + controller
cd scripts
Run a virtual shell:
pipenv shell
Install dependencies with makefile:
make
In the event of an inexplicable python error with parsing, please ensure that you are running a virtual env and have all dependencies loaded.
Run the express server:
cd ../server && npm start
If needed, run the react client:
cd ../client && npm start
The script expects three args: <tag>
<pages>
<destination>
The tag is self-explanatory.
The max amount of pages allowed by Discord is 50
.
The destination can be: server
to return to the express server, json
to print to terminal in a nice format, or mongodb
to directly insert into the database.
For example:
python3 scrapy.py buffy 2 json
Also noteworthy is that if you try to run a scraper from AWS Lambda, your generated IP from AWS's known range will be blocked even. If you know of a workaround to this, please let me know here.
GET
to http://localhost:3333/search
for an array of all tags.
POST
to http://localhost:3333/tags
and in your body, pleae include up to 5 inclusion and exclusion tags:
body: JSON.stringify({
tag1: tag1,
tag2: tag2,
tag3: tag3,
tag4: tag4,
tag5: tag5,
exclude1: exclude1,
exclude2: exclude2,
exclude3: exclude3,
exclude4: exclude4,
exclude5: exclude5,
}),
In the response, you will receive an array of JSON
:
[
{
'Search Tag': 'buffy',
'Server Name': 'Buffyverse',
'Members Online': '208',
'Creation Date': 'datetime.datetime(2019, 4, 18, 7, 33, 50, 504000',
'Invite Link': 'https://disboard.org/server/568338386272780314',
'Tag 1': 'angel',
'Tag 2': 'buffy-the-vampire-slayer',
'Tag 3': 'buffy',
'Tag 4': 'btvs',
'Tag 5': 'buffyverse'
},
{
'Search Tag': 'buffy',
'Server Name': 'Slayerverse',
'Members Online': '82',
'Creation Date': 'datetime.datetime(2022, 1, 16, 3, 1, 6, 13000',
'Invite Link': 'https://disboard.org/server/932107165685137468',
'Tag 1': 'supernatural',
'Tag 2': 'angel',
'Tag 3': 'buffy-the-vampire-slayer',
'Tag 4': 'buffy',
'Tag 5': 'buffyverse'
}
]
-
Inner single quotes (apostrophes) have been removed; you will instead see an empty space where there should be a single quote. For example, instead of the string
"Devil's Land"
, you will receive"Devil s Land"
. Parse accordingly. -
datetime
has been left as is. Import and parse accordingly. -
The current mongo collection in use is found at
TheClusterName - discord.test_servers
If you get a weird python syntax error from the scraper script, check that you have:
- run the virtual env in the script directory
- loaded all dependencies for the script
- run the server in the same shell as the virtual env
Note: Logs can be found in server/app.log
Please let me know at: [email protected]
and include screenshots of the logs and other relevant details.