Skip to content

Commit

Permalink
When testing if a blob is indexable, check size first
Browse files Browse the repository at this point in the history
Otherwise, charlock_holmes will allocate another large binary
buffer for testing the encoding, which is a problem if the binary
blob is many hundreds of MB large. It'll just fail and crash ruby.
  • Loading branch information
Scott J. Goldman committed Sep 1, 2012
1 parent 6ec907a commit e415a13
Show file tree
Hide file tree
Showing 3 changed files with 11 additions and 3 deletions.
1 change: 1 addition & 0 deletions github-linguist.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Gem::Specification.new do |s|
s.add_dependency 'escape_utils', '~> 0.2.3'
s.add_dependency 'mime-types', '~> 1.19'
s.add_dependency 'pygments.rb', '>= 0.2.13'
s.add_development_dependency 'mocha'
s.add_development_dependency 'json'
s.add_development_dependency 'rake'
s.add_development_dependency 'yajl-ruby'
Expand Down
6 changes: 3 additions & 3 deletions lib/linguist/blob_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,9 @@ def generated?
#
# Return true or false
def indexable?
if binary?
if size > 100 * 1024
false
elsif binary?
false
elsif extname == '.txt'
true
Expand All @@ -260,8 +262,6 @@ def indexable?
false
elsif generated?
false
elsif size > 100 * 1024
false
else
true
end
Expand Down
7 changes: 7 additions & 0 deletions test/test_blob.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
require 'linguist/samples'

require 'test/unit'
require 'mocha'
require 'mime/types'
require 'pygments'

Expand Down Expand Up @@ -261,6 +262,12 @@ def test_indexable
assert !blob("Text/dump.sql").indexable?
assert !blob("Binary/github.po").indexable?
assert !blob("Binary/linguist.gem").indexable?

# large binary blobs should fail on size check first, not call
# into charlock_holmes and alloc big buffers for testing encoding
b = blob("Binary/octocat.ai")
b.expects(:binary?).never
assert !b.indexable?
end

def test_language
Expand Down

0 comments on commit e415a13

Please sign in to comment.