The web crawler is written in python and is a case study project in Udacity Introduction to Python Programming
The crawler starts with a random Wikipedia page and always dig the first link in the main content! The stop conditions are:
- Arrive at the Wikipedia page for Philosophy.
- Arrive a page without any link.
- Arrive a page that has been searched for the current run. (That is in a cycle search. a -> b -> c -> a)
- Search more than 25 links.