- I thought about creating this website for combining data related to the bachelor project from the different websites (the
Student intranet
and theMET Website
) since theStudent Intranet
contains all the valid thesis list but without their respective descriptions, and to organize this data in several ways in order to help me get a full idea about all the thesis.
- The data is collected using
web scraping
using the Puppeteer and Cheerio libraries in Node.js.
Note: Authentication credentials will only be inserted inside the environmental variables file (.env) and will be used by the
htmnl
andPuppeteer
libraries for authentication.
- Therefore, they only reside on your local machine and are not sent anywhere.
- No data will be fetched unless authentication is successful.
- Step 1: Clone the repository.
- Step 2: Open the backend folder > then open the .env file and insert your username and password beside their respective variables that are already there. This is how the latter part of the file should look like.
Note:
I tried to echo these two lines, and it didn't really work, so I do it manually.
USER_NAME=<insert_your_username>
PASSWORD=<insert_your_password>
- Step 3: Open a new terminal.
- Step 4: Navigate into the backend directory.
cd backend
- Step 5: Install the necessary dependencies found inside the package.json file.
npm install
- Step 6: Start the server, which scrapes the data and stores it on your machine as .json files.
Note:
The console will print Done! when all data is fetched.
npm start
- Step 7: Open a new terminal (without closing the first one).
- Step 8: Navigate into the frontend directory.
cd frontend
- Step 9: Install the necessary dependencies found inside the package.json file.
npm install
- Step 10: Start the react app where the page will automatically open on your browser on port 3000. If it doesn't, type
localhost:3000
on the url bar.
npm start
1. View list of thesis.
- The thesis list contains information about the:
- Thesis name and supervisor as indicated in the
Student Intranet
. - As well as the category, supervisor name (as written in MET, since some thesis topics have multiple supervisors) and assistant supervisor names as indicated in the
MET website
.
- Thesis name and supervisor as indicated in the
- Any thesis that is not found in the MET website is highlighted in gray.
2. View thesis description by clicking on each expandable/collapsable table row.
3. View a separate list for all thesis that are not currently found in MET.
4. Sort both thesis list according to different criteria.
- Category Ascending.
- Category Descending.
- Supervisor Ascending.
- Supervisor Descending.
- Thesis Name Ascending.
- Thesis Name Descending.
- ID Ascending.
- ID Descending.
5. View thesis per category.
6. View thesis per supervisor.
7. View general statistics.
- Number of thesis per category.
- Number of thesis per supervisor.
- Total number of thesis.
- Number of thesis not found in MET.