Download any website from the Internet Archive Wayback Machine.
You need to install Ruby on your system (>= 1.9.2) - if you don't already have it. Then run:
gem install wayback_machine_downloader
Tip: If you run into permission errors, you might have to add sudo in front of this command.
Run wayback_machine_downloader with the base url of the website you want to retrieve as a parameter (e.g., http://example.com):
wayback_machine_downloader http://example.com
It will download the last version of every file present on Wayback Machine to websites/example.com/. It will also re-create a directory structure and auto-create index.html pages to work seamlessly with Apache and Nginx. All files downloaded are the original ones and not Wayback Machine rewritten versions. This way, URLs and links structure are the same than before.
You may want to supply a specific timestamp to lock your backup to an older version of the website, which can be found inside the urls of the regular Wayback Machine website (e.g., http://web.archive.org/web/20060716231334/http://example.com). Wayback Machine Downloader will then fetch only file versions on or prior to the timestamp specified:
wayback_machine_downloader http://example.com --timestamp 20060716231334
You may want to retrieve files that are of certain type (e.g., .pdf, .jpg, .wrd...) or are in a specific directory. To do so, you can supply the --only flag with a string or a Regex to limit what Wayback Machine Downloader will download.
wayback_machine_downloader http://example.com --only \.pdf
Contributions are welcome! Just submit a pull request via GitHub.
To run the tests:
bundle install
bundle exec rake test