Merge remote-tracking branch 'origin/master' into implementation-name

* origin/master: (57 commits)
  tweaks
  Call out CSS issues
  Fix link to linguist
  Release 1.3.3
  The command is POSIX, not Posix
  Release 1.3.2
  Spruce up old tests
  Add empty `a` to support contents with sectnums
  Add comments to describe these calls
  No longer true
  Restore "test" as the default rake task
  point to other places before creating issues
  Extconf hack no longer necessary
  Don't test jdk7
  Fallback on open3 if posix-spawn is unavailable.
  Define CommandImplementation#execute conditional on presence of Posix::Spawn
  Get html_equal tests running with minitest
  Updated deprecated .exists? in markup.rb
  Added the link to Pod::Simple on CPAN.
  Fix pod output
  ...

Conflicts:
	lib/github/markup.rb
	test/markup_test.rb

Author: bkeepers

.travis.yml

@@ -1,8 +1,11 @@
+language: ruby
 before_install: sudo pip install docutils
 rvm:
   - 1.9.3
   - 2.0.0
   - 2.1.1
+  - jruby-19mode
+jdk:
+  - oraclejdk8
 notifications:
-  disabled: true
-bundler_args: --without development
+  email: false

CONTRIBUTING.md

@@ -1,6 +1,16 @@
 # Contributing
 
-Want to contribute? Great!
+This library's only job is to decide which markup format to use and call out to an external library to convert the markup to HTML (see the [README](README.md) for more information on how markup is rendered on GitHub.com).
+
+If you are having an issue with:
+
+* **Syntax highlighting** - see [github/linguist](https://github.com/github/linguist/blob/master/CONTRIBUTING.md#fixing-syntax-highlighting)
+* **Markdown on GitHub** - contact [email protected]
+* **Styling issues on GitHub** - contact [email protected]
+
+Anything else - [search open issues](https://github.com/github/markup/issues) or create an issue and and we'll help point you in the right direction.
+
+## Submitting a Pull Request
 
 1. Fork it.
 2. Create a branch (`git checkout -b my_markup`)
@@ -9,57 +19,12 @@ Want to contribute? Great!
 5. Open a [Pull Request][1]
 6. Enjoy a refreshing Diet Coke and wait
 
-
-There are two ways to add markups.
-
-### Commands
-
-If your markup is in a language other than Ruby, drop a translator
-script in `lib/github/commands` which accepts input on STDIN and
-returns HTML on STDOUT. See [rest2html][r2h] for an example.
-
-Once your script is in place, edit `lib/github/markups.rb` and tell
-GitHub Markup about it. Again we look to [rest2html][r2hc] for
-guidance:
-
-    command(:rest2html, /re?st(.txt)?/)
-
-Here we're telling GitHub Markup of the existence of a `rest2html`
-command which should be used for any file ending in `rest`,
-`rst`, `rest.txt` or `rst.txt`. Any regular expression will do.
-
-Finally add your [tests](#testing).
-
-### Classes
-
-If your markup can be translated using a Ruby library, that's
-great. Check out `lib/github/markups.rb` for some
-examples. Let's look at Markdown:
-
-    markup(:markdown, /md|mkdn?|markdown/) do |content|
-      Markdown.new(content).to_html
-    end
-
-We give the `markup` method three bits of information: the name of the
-file to `require`, a regular expression for extensions to match, and a
-block to run with unformatted markup which should return HTML.
-
-If you need to monkeypatch a RubyGem or something, check out the
-included RDoc example.
-
-Finally add your [tests](#testing).
-
-### Testing
+## Testing
 
 To run the tests:
 
     $ rake
 
-When adding support for a new markup library, create a `README.extension` in `test/markups` along with a `README.extension.html`. As you may imagine, the `README.extension` should be your known input and the
-`README.extension.html` should be the desired output.
-
-Now run the tests: `rake`
-
 If nothing complains, congratulations!
 
 ## Releasing a new version

Gemfile

@@ -1,10 +1,13 @@
 source "http://rubygems.org"
 gemspec
-gem "redcarpet"
+
+gem "posix-spawn", :platforms => :ruby
+gem "redcarpet", :platforms => :ruby
+gem "kramdown", :platforms => :jruby
 gem "RedCloth"
 gem "rdoc", "~>3.6"
-gem "org-ruby", "= 0.9.1.gh"
+gem "org-ruby", "= 0.9.9"
 gem "creole", "~>0.3.6"
-gem "wikicloth", "=0.8.1"
+gem "wikicloth", "=0.8.1", :platforms => :ruby
 gem "asciidoctor", "= 0.1.4"
-gem "rake"
+gem "rake"

HISTORY.md

@@ -1,3 +1,22 @@
+## 1.3.3 (2015-02-17)
+
+* Address a slight typo with `POSIX` [#456](https://github.com/github/markup/pull/456)
+
+[Full changelog](https://github.com/github/markup/compare/v1.3.2...v1.3.3)
+
+## 1.3.2 (2015-02-17)
+
+* RST: Output code instead of tt for inline literals [#370](https://github.com/github/markup/pull/370)
+* RST: Add IDs to headers so that `.. contents` works with `.. sectnum` [#391](https://github.com/github/markup/pull/391)
+
+[Full changelog](https://github.com/github/markup/compare/v1.3.1...v1.3.2)
+
+## 1.3.1 (2014-11-13)
+
+* Fix name error when trying to use newer versions of RedCarpet [#387](https://github.com/github/markup/pull/387)
+
+[Full changelog](https://github.com/github/markup/compare/v1.3.0...v1.3.1)
+
 ## 1.3.0 (2014-09-11)
 
 * Extend the field limit for tables to 50 characters for RST [#306](https://github.com/github/markup/pull/306)

README.md

@@ -1,8 +1,15 @@
 GitHub Markup
 =============
 
-We use this library on GitHub when rendering your README or any other
-rich text file. The generated HTML is then run through filters in the [html-pipeline](https://github.com/jch/html-pipeline) to perform things like [sanitization](#html-sanitization) and [syntax highlighting](https://github.com/jch/html-pipeline/blob/master/lib/html/pipeline/syntax_highlight_filter.rb).
+This library is the first step of a journey that every markup file in a repository goes on before it is rendered on GitHub.com:
+
+0. This library converts the raw markup to HTML. See the list of [supported markup formats](#markups) below.
+0. The HTML is sanitized, aggressively removing things that could harm you and your kin—such as `script` tags, inline-styles, and `class` or `id` attributes. See the [sanitization filter](https://github.com/jch/html-pipeline/blob/master/lib/html/pipeline/sanitization_filter.rb) for the full whitelist.
+0. Syntax highlighting is performed on code blocks. See [github/linguist](https://github.com/github/linguist#syntax-highlighting) for more information about syntax highlighting.
+0. The HTML is passed through other filters in the [html-pipeline](https://github.com/jch/html-pipeline) that add special sauce, such as [emoji](https://github.com/jch/html-pipeline/blob/master/lib/html/pipeline/emoji_filter.rb), [task lists](https://github.com/github/task_list/blob/master/lib/task_list/filter.rb), [named anchors](https://github.com/jch/html-pipeline/blob/master/lib/html/pipeline/toc_filter.rb), [CDN caching for images](https://github.com/jch/html-pipeline/blob/master/lib/html/pipeline/camo_filter.rb), and  [autolinking](https://github.com/jch/html-pipeline/blob/master/lib/html/pipeline/autolink_filter.rb).
+0. The resulting HTML is rendered on GitHub.com.
+
+Please see our [contributing guidelines](CONTRIBUTING.md) before reporting an issue.
 
 Markups
 -------
@@ -19,7 +26,7 @@ you wish to run the library. You can also run `script/bootstrap` to fetch them a
 * [.rst](http://docutils.sourceforge.net/rst.html) -- `easy_install docutils`
 * [.asciidoc, .adoc, .asc](http://asciidoc.org/) -- `gem install asciidoctor` (http://asciidoctor.org)
 * [.pod](http://search.cpan.org/dist/perl/pod/perlpod.pod) -- `Pod::Simple::HTML`
-  comes with Perl >= 5.10. Lower versions should install Pod::Simple from CPAN.
+  comes with Perl >= 5.10. Lower versions should install [Pod::Simple](http://search.cpan.org/~dwheeler/Pod-Simple-3.28/lib/Pod/Simple.pod) from CPAN.
 
 Installation
 -----------
@@ -41,32 +48,3 @@ Contributing
 ------------
 
 See [Contributing](CONTRIBUTING.md)
-
-HTML sanitization
------------------
-
-HTML rendered by the various markup language processors gets passed through an [HTML sanitization filter](https://github.com/jch/html-pipeline/blob/master/lib/html/pipeline/sanitization_filter.rb) for security reasons. HTML elements not in the whitelist are removed. HTML attributes not in the whitelist are removed from the preserved elements.
-
-The following HTML elements, organized by category, are whitelisted:
-
-|Type | Elements
-|------|----------
-|Headings | `h1`, `h2`, `h3`, `h4`, `h5`, `h6`, `h7`, `h8`
-|Prose |  `p`, `div`, `blockquote`
-|Formatted | `pre`
-| Inline | `b`, `i`, `strong`, `em`, `tt`, `code`, `ins`, `del`, `sup`, `sub`, `kbd`, `samp`, `q`, `var`
-| Lists | `ol`, `ul`, `li`, `dl`, `dt`, `dd`
-| Tables | `table`, `thead`, `tbody`, `tfoot`, `tr`, `td`, `th`
-| Breaks | `br`, `hr`
-| Ruby (East Asian) | `ruby`, `rt`, `rp`
-
-The following attributes, organized by element, are whitelisted:
-
-|Element | Attributes
-|------|----------
-| `a` | `href` (`http://`, `https://`, `mailto://`, `github-windows://`, and `github-mac://` URI schemes and relative paths only)
-| `img` | `src` (`http://` and `https://` URI schemes and relative paths only)
-| `div` | `itemscope`, `itemtype`
-| All | `abbr`, `accept`, `accept-charset`, `accesskey`, `action`, `align`, `alt`, `axis`, `border`, `cellpadding`, `cellspacing`, `char`, `charoff`, `charset`, `checked`, `cite`, `clear`, `cols`, `colspan`, `color`, `compact`, `coords`, `datetime`, `dir`, `disabled`, `enctype`, `for`, `frame`, `headers`, `height`, `hreflang`, `hspace`, `ismap`, `label`, `lang`, `longdesc`, `maxlength`, `media`, `method`, `multiple`, `name`, `nohref`, `noshade`, `nowrap`, `prompt`, `readonly`, `rel`, `rev`, `rows`, `rowspan`, `rules`, `scope`, `selected`, `shape`, `size`, `span`, `start`, `summary`, `tabindex`, `target`, `title`, `type`, `usemap`, `valign`, `value`, `vspace`, `width`, `itemprop`
-
-Note that the `id` attribute is *not* whitelisted.

github-markup.gemspec

@@ -16,5 +16,9 @@ Gem::Specification.new do |s|
   s.test_files    = s.files.grep(%r{^(test|spec|features)/})
   s.require_paths = %w[lib]
 
-  s.add_dependency "posix-spawn", "~> 0.3.8"
+  s.add_development_dependency 'minitest', '~> 5.4.3'
+  s.add_development_dependency 'html-pipeline', '~> 1.0'
+  s.add_development_dependency 'sanitize', '~> 2.1.0'
+  s.add_development_dependency 'nokogiri', '~> 1.6.1'
+  s.add_development_dependency 'nokogiri-diff', '~> 0.2.0'
 end

lib/github-markup.rb

@@ -1,6 +1,6 @@
 module GitHub
   module Markup
-    VERSION = '1.3.0'
+    VERSION = '1.3.3'
     Version = VERSION
   end
 end

lib/github/commands/foo.rst

@@ -66,12 +66,15 @@ class GitHubHTMLTranslator(HTMLTranslator):
     # see also: http://bit.ly/1exfq2h (warning! sourceforge link.)
     def depart_document(self, node):
         HTMLTranslator.depart_document(self, node)
-        self.html_body.pop(0)
-        self.html_body.pop()
+        self.html_body.pop(0) # pop the starting <div> off
+        self.html_body.pop() # pop the ending </div> off
 
     # technique for visiting sections, without generating additional divs
     # see also: http://bit.ly/NHtyRx
+    # the a is to support ::contents with ::sectnums: http://git.io/N1yC
     def visit_section(self, node):
+        id_attribute = node.attributes['ids'][0]
+        self.body.append('<a name="%s"></a>\n' % id_attribute)
         self.section_level += 1
 
     def depart_section(self, node):
@@ -86,6 +89,14 @@ class GitHubHTMLTranslator(HTMLTranslator):
         else:
             self.body.append(self.starttag(node, 'pre'))
 
+    # always wrap two-backtick rst inline literals in <code>, not <tt>
+    # this also avoids the generation of superfluous <span> tags
+    def visit_literal(self, node):
+        self.body.append(self.starttag(node, 'code', suffix=''))
+
+    def depart_literal(self, node):
+        self.body.append('</code>')
+
     def visit_table(self, node):
         classes = ' '.join(['docutils', self.settings.table_style]).strip()
         self.body.append(

lib/github/commands/rest2html

@@ -31,7 +31,7 @@ def markup(file, pattern, opts = {}, &block)
     end
 
     def command(command, regexp, name, &block)
-      if File.exists?(file = File.dirname(__FILE__) + "/commands/#{command}")
+      if File.exist?(file = File.dirname(__FILE__) + "/commands/#{command}")
         command = file
       end
 

lib/github/markup.rb

@@ -1,4 +1,9 @@
-require "posix-spawn"
+begin
+  require "posix-spawn"
+rescue LoadError
+  require "open3"
+end
+
 require "github/markup/implementation"
 
 module GitHub
@@ -33,14 +38,34 @@ def call_block(rendered, content)
         end
       end
 
-      def execute(command, target)
-        spawn = POSIX::Spawn::Child.new(*command, :input => target)
-        if spawn.status.success?
-          spawn.out.gsub("\r", '').force_encoding(target.encoding)
-        else
-          raise CommandError.new(spawn.err.strip)
+      if defined?(POSIX::Spawn)
+        def execute(command, target)
+          spawn = POSIX::Spawn::Child.new(*command, :input => target)
+          if spawn.status.success?
+            sanitize(spawn.out, target.encoding)
+          else
+            raise CommandError.new(spawn.err.strip)
+          end
+        end
+      else
+        def execute(command, target)
+          output = Open3.popen3(*command) do |stdin, stdout, stderr, wait_thr|
+            stdin.puts target
+            stdin.close
+            if wait_thr.value.success?
+              stdout.readlines
+            else
+              raise CommandError.new(stderr.readlines.join('').strip)
+            end
+          end
+          sanitize(output.join(''), target.encoding)
         end
       end
+
+      def sanitize(input, encoding)
+        input.gsub("\r", '').force_encoding(encoding)
+      end
+
     end
   end
 end

lib/github/markup/command_implementation.rb

@@ -8,7 +8,7 @@ class Markdown < Implementation
           GitHub::Markdown.render(content)
         },
         "redcarpet" => proc { |content|
-          RedcarpetCompat.new(content).to_html
+          Redcarpet::Markdown.new(Redcarpet::Render::HTML).render(content)
         },
         "rdiscount" => proc { |content|
           RDiscount.new(content).to_html

lib/github/markup/markdown.rb

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+echo "failure message">&2 && false