Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importing large Tumblr site, running into import errors #333

Closed
readtedium opened this issue Dec 25, 2017 · 1 comment
Closed

Importing large Tumblr site, running into import errors #333

readtedium opened this issue Dec 25, 2017 · 1 comment

Comments

@readtedium
Copy link

Hi there, I'm looking to move my Tumblr content to a static site for archival purposes. Unfortunately, that's harder than it sounds because I have more than 20,000 posts. I ran into an issue where the importer would stop because the pictures from one of my old posts were missing from the Tumblr servers, but having removed that old post I'm now running into another error:

/usr/local/Cellar/ruby/2.4.3_1/lib/ruby/2.4.0/open-uri.rb:363:in `open_http': 429 Limit Exceeded (OpenURI::HTTPError)
from /usr/local/Cellar/ruby/2.4.3_1/lib/ruby/2.4.0/open-uri.rb:741:in `buffer_open'
from /usr/local/Cellar/ruby/2.4.3_1/lib/ruby/2.4.0/open-uri.rb:212:in `block in open_loop'
from /usr/local/Cellar/ruby/2.4.3_1/lib/ruby/2.4.0/open-uri.rb:210:in `catch'
from /usr/local/Cellar/ruby/2.4.3_1/lib/ruby/2.4.0/open-uri.rb:210:in `open_loop'
from /usr/local/Cellar/ruby/2.4.3_1/lib/ruby/2.4.0/open-uri.rb:151:in `open_uri'
from /usr/local/Cellar/ruby/2.4.3_1/lib/ruby/2.4.0/open-uri.rb:721:in `open'
from /usr/local/Cellar/ruby/2.4.3_1/lib/ruby/2.4.0/open-uri.rb:35:in `open'
from /usr/local/lib/ruby/gems/2.4.0/gems/jekyll-import-0.13.0/lib/jekyll-import/importers/tumblr.rb:43:in `process'
from /usr/local/lib/ruby/gems/2.4.0/gems/jekyll-import-0.13.0/lib/jekyll-import/importer.rb:23:in `run'
from -e:2:in `<main>'

I'm going to try and restart the process in a fresh directory, as I got further before the image problem cropped up (11,000 posts) than I did before this error appeared (7,000 posts). However, If there's any suggestions as to what to do to handle this to help slow down the HTTP requests so that I can avoid this problem, I'd be greatly appreciative. Thanks.

@readtedium
Copy link
Author

Got it working—the loading issue might be worth looking into, but downloading all the images from Tumblr had a side effect of slowing down the API enough that I didn't run into the error.

@jekyll jekyll locked and limited conversation to collaborators Apr 30, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants