Catch JSON decode errors in oembed responses (wagtail#6648)

Fixes wagtail#6646 - for private videos, Youtube's oembed endpoint returns the text "Forbidden", but with a 200 status code, which fails JSON decoding.

CHANGELOG.txt

@@ -12,6 +12,7 @@ Changelog
  * Support returning None from `register_page_action_menu_item` and `register_snippet_action_menu_item` to skip registering an item (Vadim Karpenko)
  * Fix: Stop menu icon overlapping the breadcrumb on small viewport widths in page editor (Karran Besen)
  * Fix: Make sure document chooser pagination preserves the selected collection when moving between pages (Alex Sa)
+ * Fix: Gracefully handle oEmbed endpoints returning non-JSON responses (Matt Westcott)
 
 
 2.11.3 (10.12.2020)

docs/releases/2.12.rst

@@ -31,6 +31,7 @@ Bug fixes
 
  * Stop menu icon overlapping the breadcrumb on small viewport widths in page editor (Karran Besen)
  * Make sure document chooser pagination preserves the selected collection when moving between pages (Alex Sa)
+ * Gracefully handle oEmbed endpoints returning non-JSON responses (Matt Westcott)
 
 
 Upgrade considerations

wagtail/embeds/finders/oembed.py

@@ -60,9 +60,9 @@ def find_embed(self, url, max_width=None):
         request.add_header('User-agent', 'Mozilla/5.0')
         try:
             r = urllib_request.urlopen(request)
-        except URLError:
+            oembed = json.loads(r.read().decode('utf-8'))
+        except (URLError, json.decoder.JSONDecodeError):
             raise EmbedNotFoundException
-        oembed = json.loads(r.read().decode('utf-8'))
 
         # Convert photos into HTML
         if oembed['type'] == 'photo':

wagtail/embeds/tests/test_embeds.py

@@ -367,6 +367,14 @@ def test_oembed_invalid_request(self):
             self.assertRaises(EmbedNotFoundException, OEmbedFinder().find_embed,
                               "http://www.youtube.com/watch/")
 
+    @patch('urllib.request.urlopen')
+    def test_oembed_non_json_response(self, urlopen):
+        urlopen.return_value = self.dummy_response
+        self.assertRaises(
+            EmbedNotFoundException, OEmbedFinder().find_embed,
+            "https://www.youtube.com/watch?v=ReblZ7o7lu4"
+        )
+
     @patch('urllib.request.urlopen')
     @patch('json.loads')
     def test_oembed_photo_request(self, loads, urlopen):