Upgrade Instructions

====
   * Uninstall the python Cassandra package that we previously depended on (it has a namespace conflict with the new package we depend on).  To find it:

     $ python -c "import cassandra; print cassandra.__file__"

    and rm -r the .egg directory after "site-packages/"

   * This version relies on cython, so if "make" fails, you may have to install cython via your distro's package manager.

     $ cd reddit/r2
     $ python setup.py develop # possibly with "sudo" depending on your install
     $ make

   * Cassandra is now required for the caching layer.  An example storage-conf.xml can be found in reddit/srv/cassandra.  Make sure that the additional <Keyspace> items are included in your conf file.

   * remove the query_queue_reader services if they are running.  add new gen_time_listings.sh instead.  Suggested cron:

    0    */2 *   *   *   $SCRIPTS/gen_time_listings.sh year '("month","year")'
    */5  *   *   *   *   $SCRIPTS/gen_time_listings.sh week '("day","week")'
    *    *   *   *   *   $SCRIPTS/gen_time_listings.sh hour '("hour",)'

   where $SCRIPTS is the location of this script directory

Features and Bugfixes
====
 * Mobile reddit:
   * templates are in r2/templates of the form *.compact
   * css is r2/public/static/css/compact.css
     * beginning of a sass-based (http://sass-lang.com/) compact.scss
   * reachable via .compact extension or from the "i" subdomain.
 * Cassandra is now *required*, and votes are currently written out to both cassandra and postgres (part of an eventual migration).
 * attempt to make the db connection code a little smarter.
   * A dropped DB connection will mark the connection dead and randomly attempt to reconnect.
   * A dropped db connection on start will permanently mark the connection as dead.
 * Calculate the time-filtered top/controversy listings using mapreduce instead of prec_links (new cron job in reddit/scripts)
 * allow default user/pass for database to be specified with '*' to fallback on db_user and db_pass in the INI file
 * Search feedback buttons
 * make deleted comments not show up in your inbox.
 * move last_visited into cassandra
 * Swallow rare, race-conditiony POST_save/hide/subscribe problems
 * Apparently we haven't been breaking properly for the past few weeks.

.gitignore

@@ -31,3 +31,13 @@ r2/myproduction.ini
 .DS_Store
 r2/r2.egg-info/**
 r2/r2/public/static/sprite.png
+r2/_builder.egg-info/
+r2/_normalized_hot.egg-info/
+r2/_sorts.egg-info/
+r2/r2/lib/_normalized_hot.c
+r2/r2/lib/db/_sorts.c
+r2/r2/lib/sgm.c
+r2/r2/lib/wrapped.c
+r2/r2/models/_builder.c
+r2/sgm.egg-info/
+r2/wrapped.egg-info/

r2/Makefile

@@ -21,10 +21,10 @@
 ################################################################################
 
 # Jacascript files to be compressified
-js_targets  = jquery.js jquery.json.js jquery.reddit.js reddit.js ui.core.js ui.datepicker.js sponsored.js jquery.flot.js jquery.lazyload.js
+js_targets  = jquery.js jquery.json.js jquery.reddit.js reddit.js ui.core.js ui.datepicker.js sponsored.js jquery.flot.js compact.js
 # CSS targets 
 main_css = reddit.css
-css_targets = reddit-ie6-hax.css reddit-ie7-hax.css mobile.css spreadshirt.css
+css_targets = reddit-ie6-hax.css reddit-ie7-hax.css mobile.css spreadshirt.css compact.css
 
 SED=sed
 CAT=cat
@@ -47,6 +47,9 @@ CSSTARGETS := $(foreach css, $(css_targets), $(static_dir)/$(css))
 MAINCSS    := $(foreach css, $(main_css), $(static_dir)/$(css))
 RTLCSS      = $(CSSTARGETS:.css=-rtl.css) $(MAINCSS:.css=-rtl.css)
 
+PYX_FILES := $(shell find . -name \*.pyx)
+PYXSO_FILES := $(PYX_FILES:.pyx=.so)
+
 
 MD5S = $(JSTARGETS:=.md5) $(CSSTARGETS:=.md5) $(MAINCSS:=.md5) $(RTLCSS:=.md5)
 
@@ -63,7 +66,7 @@ else
 endif
 
 
-all:	$(JSTARGETS) $(CSSTARGETS) $(MD5S) $(RTLCSS) $(INIS)
+all:	$(JSTARGETS) $(CSSTARGETS) $(MD5S) $(RTLCSS) $(INIS) $(PYXSO_FILES)
 
 .PHONY: 	js css md5 rtl clean all
 
@@ -85,7 +88,12 @@ $(RTLCSS):	%-rtl.css : %.css
 	   -e "s/>####</right/g" \
 	   -e "s/\(margin\|padding\):\s*\([^; ]\+\)\s\+\([^; ]\+\)\s\+\([^; ]\+\)\s\+\([^; ]\+\)/\1:\2 \5 \4 \3/g" $< > $@
 
-
+$(PYXSO_FILES): %.so : %.pyx
+	rm -f $(<:.pyx=.pyc)
+	cython $<
+	mkdir -p tmp
+	PYTHONPATH=tmp python pyx_setup.py develop -d tmp -b tmp $<
+	rm -r tmp
 
 
 js: 		$(JSTARGETS)

r2/example.ini

@@ -91,7 +91,7 @@ servicecaches = 127.0.0.1:11211
 # memcaches that sit in front of cassandra
 permacache_memcaches = 127.0.0.1:11211
 # cassandra hosts. one of these will be chosen at random by pycassa
-cassandra_seeds =
+cassandra_seeds = 127.0.0.1:9160
 
 # -- url cache options --
 url_caches = 127.0.0.1:11211
@@ -338,5 +338,5 @@ beaker.session_secret = somesecret
 # WARNING: *THE LINE BELOW MUST BE UNCOMMENTED ON A PRODUCTION ENVIRONMENT*
 # Debug mode will enable the interactive debugging tool, allowing ANYONE to
 # execute malicious code after an exception is raised.
-#set debug = false
+set debug = true
 

r2/pyx_setup.py

@@ -0,0 +1,54 @@
+#!/usr/bin/env python
+
+# The contents of this file are subject to the Common Public Attribution
+# License Version 1.0. (the "License"); you may not use this file except in
+# compliance with the License. You may obtain a copy of the License at
+# http://code.reddit.com/LICENSE. The License is based on the Mozilla Public
+# License Version 1.1, but Sections 14 and 15 have been added to cover use of
+# software over a computer network and provide for limited attribution for the
+# Original Developer. In addition, Exhibit A has been modified to be consistent
+# with Exhibit B.
+#
+# Software distributed under the License is distributed on an "AS IS" basis,
+# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
+# the specific language governing rights and limitations under the License.
+#
+# The Original Code is Reddit.
+#
+# The Original Developer is the Initial Developer.  The Initial Developer of the
+# Original Code is CondeNet, Inc.
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
+# CondeNet, Inc. All Rights Reserved.
+################################################################################
+
+from ez_setup import use_setuptools
+use_setuptools()
+from setuptools import find_packages#, setup
+from distutils.core import setup, Extension
+from Cython.Distutils import build_ext
+import os, sys
+import shutil
+
+def build_so(pyx_file):
+    if not pyx_file.endswith(".pyx"):
+        raise ValueError, "expected a pyx file, got %s" % pyx_file
+    if not os.path.exists(pyx_file):
+        raise ValueError, "pyx file does not exist: %s" % pyx_file
+    lib = pyx_file[:-4].replace('/', '.').lstrip(".")
+    name = lib.split('.')[-1]
+    ext_modules = [ Extension(lib, [pyx_file]) ]
+    setup(
+        name=name,
+        version="",
+        packages=find_packages(),
+        include_package_data=True,
+        test_suite = 'nose.collector',
+        cmdclass = {'build_ext': build_ext},
+        ext_modules = ext_modules,
+        )
+    shutil.rmtree(name + '.egg-info')
+
+
+if __name__ == "__main__":
+    build_so(sys.argv.pop())

r2/r2/config/middleware.py

@@ -9,7 +9,7 @@
 #
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
-# the specific language governing rights and limitations under the License.
+# the specific language governing rig and limitations under the License.
 #
 # The Original Code is Reddit.
 #
@@ -43,6 +43,7 @@
 
 #from pylons.middleware import error_mapper
 def error_mapper(code, message, environ, global_conf=None, **kw):
+    from pylons import c
     if environ.get('pylons.error_call'):
         return None
     else:
@@ -65,7 +66,6 @@ def error_mapper(code, message, environ, global_conf=None, **kw):
 
         #preserve x-sup-id when 304ing
         if code == 304:
-            from pylons import c
             #check to see if c is useable
             try:
                 c.test
@@ -311,6 +311,10 @@ def __call__(self, environ, start_response):
             # subdomains which change the extension
             elif sd == 'm':
                 environ['reddit-domain-extension'] = 'mobile'
+            elif sd == 'I':
+                environ['reddit-domain-extension'] = 'compact'
+            elif sd == 'i':
+                environ['reddit-domain-extension'] = 'compact'
             elif sd in ('api', 'rss', 'xml', 'json'):
                 environ['reddit-domain-extension'] = sd
             elif (len(sd) == 2 or (len(sd) == 5 and sd[2] == '-')) and self.lang_re.match(sd):
@@ -368,26 +372,30 @@ def __call__(self, environ, start_response):
 class ExtensionMiddleware(object):
     ext_pattern = re.compile(r'\.([^/]+)$')
 
-    extensions = {'rss' : ('xml', 'text/xml; charset=UTF-8'),
-                  'xml' : ('xml', 'text/xml; charset=UTF-8'),
-                  'js' : ('js', 'text/javascript; charset=UTF-8'),
-                  'wired' : ('wired', 'text/javascript; charset=UTF-8'),
-                  'embed' : ('htmllite', 'text/javascript; charset=UTF-8'),
-                  'mobile' : ('mobile', 'text/html; charset=UTF-8'),
-                  'png' : ('png', 'image/png'),
-                  'css' : ('css', 'text/css'),
-                  'csv' : ('csv', 'text/csv; charset=UTF-8'),
-                  'api' : (api_type(), 'application/json; charset=UTF-8'),
-                  'json' : (api_type(), 'application/json; charset=UTF-8'),
-                  'json-html' : (api_type('html'), 'application/json; charset=UTF-8')}
+    extensions = (('rss' , ('xml', 'text/xml; charset=UTF-8')),
+                  ('xml' , ('xml', 'text/xml; charset=UTF-8')),
+                  ('js' , ('js', 'text/javascript; charset=UTF-8')),
+                  ('wired' , ('wired', 'text/javascript; charset=UTF-8')),
+                  ('embed' , ('htmllite', 'text/javascript; charset=UTF-8')),
+                  ('mobile' , ('mobile', 'text/html; charset=UTF-8')),
+                  ('png' , ('png', 'image/png')),
+                  ('css' , ('css', 'text/css')),
+                  ('csv' , ('csv', 'text/csv; charset=UTF-8')),
+                  ('api' , (api_type(), 'application/json; charset=UTF-8')),
+                  ('json-html' , (api_type('html'), 'application/json; charset=UTF-8')),
+                  ('json-compact' , (api_type('compact'), 'application/json; charset=UTF-8')),
+                  ('compact' , ('compact', 'text/html; charset=UTF-8')),
+                  ('json' , (api_type(), 'application/json; charset=UTF-8')),
+                  ('i' , ('compact', 'text/html; charset=UTF-8')),
+                  )
 
     def __init__(self, app):
         self.app = app
 
     def __call__(self, environ, start_response):
         path = environ['PATH_INFO']
         domain_ext = environ.get('reddit-domain-extension')
-        for ext, val in self.extensions.iteritems():
+        for ext, val in self.extensions:
             if ext == domain_ext or path.endswith('.' + ext):
                 environ['extension'] = ext
                 environ['render_style'] = val[0]

r2/r2/config/routing.py

@@ -33,6 +33,7 @@ def make_map(global_conf={}, app_conf={}):
     admin_routes.add(mc)
 
     mc('/login',    controller='forms', action='login')
+    mc('/register',    controller='forms', action='register')
     mc('/logout',   controller='forms', action='logout')
     mc('/verify',   controller='forms', action='verify')
     mc('/adminon',  controller='forms', action='adminon')
@@ -53,7 +54,7 @@ def make_map(global_conf={}, app_conf={}):
 
     mc('/reddits/create', controller='front', action='newreddit')
     mc('/reddits/search', controller='front', action='search_reddits')
-    mc('/reddits/login', controller='front', action='login')
+    mc('/reddits/login', controller='forms', action='login')
     mc('/reddits/:where', controller='reddits', action='listing',
        where = 'popular',
        requirements=dict(where="popular|new|banned"))
@@ -72,6 +73,7 @@ def make_map(global_conf={}, app_conf={}):
 
     mc('/widget', controller='buttons', action='widget_demo_page')
     mc('/bookmarklets', controller='buttons', action='bookmarklets')
+    mc('/iphonebookmarklet', controller='buttons', action='iphonebookmarklets')
 
     mc('/awards', controller='front', action='awards')
 
@@ -146,6 +148,7 @@ def make_map(global_conf={}, app_conf={}):
     mc('/promoted/', controller='promoted', action = "listing",
        sort = "")
 
+    mc('/health/threads', controller='health', action='threads')
     mc('/health', controller='health', action='health')
     mc('/shutdown', controller='health', action='shutdown')
 
@@ -235,6 +238,7 @@ def make_map(global_conf={}, app_conf={}):
     mc("/ads/r/:reddit_name", controller = "ad", action = "ad")
     mc("/ads/:codename", controller = "ad", action = "ad_by_codename")
 
+    mc("/try", controller = "forms", action = "try_compact")
     mc('/comscore-iframe/', controller='mediaembed', action='comscore')
     mc('/comscore-iframe/*url', controller='mediaembed', action='comscore')
 

r2/r2/config/templates.py

@@ -27,6 +27,7 @@
 def api(type, cls):
     tpm.add_handler(type, 'api', cls())
     tpm.add_handler(type, 'api-html', cls())
+    tpm.add_handler(type, 'api-compact', cls())
 
 # blanket fallback rule
 api('templated', NullJsonTemplate)

r2/r2/controllers/api.py

@@ -19,7 +19,7 @@
 # All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
-from reddit_base import RedditController, set_user_cookie
+from reddit_base import RedditController, MinimalController, set_user_cookie
 
 from pylons.i18n import _
 from pylons import c, request
@@ -34,13 +34,12 @@
 from r2.lib.utils import timeago, tup, filter_links
 from r2.lib.pages import FriendList, ContributorList, ModList, \
     BannedList, BoringPage, FormPage, CssError, UploadedImage, \
-    ClickGadget
+    ClickGadget, UrlParser
 from r2.lib.utils.trial_utils import indict, end_trial, trial_info
 from r2.lib.pages.things import wrap_links, default_thing_wrapper
 
 from r2.lib import spreadshirt
 from r2.lib.menus import CommentSortMenu
-from r2.lib.normalized_hot import expire_hot
 from r2.lib.captcha import get_iden
 from r2.lib.strings import strings
 from r2.lib.filters import _force_unicode, websafe_json, websafe, spaceCompress
@@ -71,7 +70,7 @@ def reject_vote(thing):
               (voteword, c.user.name, request.ip, thing.__class__.__name__,
                thing._id36, request.referer), "info")
 
-class ApiminimalController(RedditController):
+class ApiminimalController(MinimalController):
     """
     Put API calls in here which won't come from logged in users (or
     don't rely on the user being logged int)
@@ -197,9 +196,10 @@ def POST_compose(self, form, jquery, to, subject, body, ip):
                    selftext = VMarkdown('text'),
                    kind = VOneOf('kind', ['link', 'self', 'poll']),
                    then = VOneOf('then', ('tb', 'comments'),
-                                 default='comments'))
+                                 default='comments'),
+                   extension = VLength("extension", 20))
     def POST_submit(self, form, jquery, url, selftext, kind, title,
-                    save, sr, ip, then):
+                    save, sr, ip, then, extension):
         from r2.models.admintools import is_banned_domain
 
         if isinstance(url, (unicode, str)):
@@ -238,7 +238,12 @@ def POST_submit(self, form, jquery, url, selftext, kind, title,
                 pass
             elif form.has_errors("url", errors.ALREADY_SUB):
                 check_domain = False
-                form.redirect(url[0].already_submitted_link)
+                u = url[0].already_submitted_link
+                if extension:
+                    u = UrlParser(u)
+                    u.set_extension(extension)
+                    u = u.unparse()
+                form.redirect(u)
             # check for title, otherwise look it up and return it
             elif form.has_errors("title", errors.NO_TEXT):
                 pass
@@ -324,7 +329,8 @@ def POST_submit(self, form, jquery, url, selftext, kind, title,
         elif then == 'tb':
             form.attr('target', '_top')
             path = add_sr('/tb/%s' % l._id36)
-
+        if extension:
+            path += ".%s" % extension
         form.redirect(path)
 
     @validatedForm(VRatelimit(rate_ip = True,
@@ -636,7 +642,6 @@ def POST_del(self, thing):
         #expire the item from the sr cache
         if isinstance(thing, Link):
             sr = thing.subreddit_slow
-            expire_hot(sr)
             queries.delete_links(thing)
 
         #comments have special delete tasks
@@ -1234,6 +1239,23 @@ def POST_site_admin(self, form, jquery, name, ip, sr,
         else:
             jquery.refresh()
 
+    @noresponse(q = VPrintable('q', max_length=500),
+                sort = VPrintable('sort', max_length=10),
+                t = VPrintable('t', max_length=10),
+                approval = VBoolean('approval'))
+    def POST_searchfeedback(self, q, sort, t, approval):
+        timestamp = c.start_time.strftime("%Y/%m/%d-%H:%M:%S")
+        if c.user_is_loggedin:
+            username = c.user.name
+        else:
+            username = None
+        d = dict(username=username, q=q, sort=sort, t=t)
+        hex = md5(repr(d)).hexdigest()
+        key = "searchfeedback-%s-%s-%s" % (timestamp[:10], request.ip, hex)
+        d['timestamp'] = timestamp
+        d['approval'] = approval
+        g.hardcache.set(key, d, time=86400 * 7)
+
     @noresponse(VUser(), VModhash(),
                 why = VSrCanBan('id'),
                 thing = VByName('id'))
@@ -1530,25 +1552,21 @@ def POST_subscribe(self, action, sr):
             self._subscribe(sr, action == 'sub')
 
     def _subscribe(self, sr, sub):
-        Subreddit.subscribe_defaults(c.user)
-
-        sub_key = "subscription-%s-%s" % (c.user.name, sr.name)
+        try:
+            Subreddit.subscribe_defaults(c.user)
 
-        if sub:
-            try:
+            if sub:
                 if sr.add_subscriber(c.user):
                     sr._incr('_ups', 1)
-            except CreationError:
-                # This only seems to happen when someone is pounding on the
-                # subscribe button or the DBs are really lagged; either way,
-                # some other proc has already handled this subscribe request.
-                return
-        else:
-            if sr.remove_subscriber(c.user):
-                g.cache.delete(sub_key)
-                sr._incr('_ups', -1)
-        changed(sr)
-
+            else:
+                if sr.remove_subscriber(c.user):
+                    sr._incr('_ups', -1)
+            changed(sr)
+        except CreationError:
+            # This only seems to happen when someone is pounding on the
+            # subscribe button or the DBs are really lagged; either way,
+            # some other proc has already handled this subscribe request.
+            return
 
     @noresponse(VAdmin(),
                 tr = VTranslation("id"))