Lucas is a web crawler built using Go and the Colly library.
It is currently setup to crawl floryday -> it will write its output to the connected psql DB and output the results of its latest crawl to the console
Note: Environmental variables for DB setup are included in the .env file.
cd db
docker-compose up -d
psql -h localhost -U user lucas_db -f dbsetup.sql
# note if using docker toolbox you may prefer the following cmd
psql -h "toolbox-ip-address" -p "5432" -U "user" -d "lucas_db" -f "dbsetup.sql"
go.mod is used for dependency management.
The main package is located under src/app -> with the main function located within lucas.go. To run the scraper:
go run src/app/*.go
To run Lucas in a Docker container
# build docker image
docker build src/app
# run docker container and portforward port 3000
docker run -ti -p 8000:8000 --network="host" <docker-image-id>
# publish docker image to docker hub
docker push <docker-repo>