Merge pull request #1671 from jeffwidman/support-json-top-level-arrays

Add support for serializing top-level arrays to JSON

AUTHORS

@@ -15,9 +15,11 @@ Patches and Suggestions
 - Chris Grindstaff
 - Christopher Grebs
 - Daniel Neuhäuser
+- David Lord @davidism
 - Edmond Burnett
 - Florent Xicluna
 - Georg Brandl
+- Jeff Widman @jeffwidman
 - Justin Quick
 - Kenneth Reitz
 - Keyan Pishdadian
@@ -32,4 +34,3 @@ Patches and Suggestions
 - Stephane Wirtel
 - Thomas Schranz
 - Zhao Xiaohong
-- David Lord @davidism

CHANGES

@@ -8,6 +8,9 @@ Version 1.0
 
 (release date to be announced, codename to be selected)
 
+- Added support to serializing top-level arrays to :func:`flask.jsonify`. This
+  introduces a security risk in ancient browsers. See
+  :ref:`json_security` for details.
 - Added before_render_template signal.
 - Added `**kwargs` to :meth:`flask.Test.test_client` to support passing
   additional keyword arguments to the constructor of

docs/security.rst

@@ -95,81 +95,12 @@ the form validation framework, which does not exist in Flask.
 JSON Security
 -------------
 
-.. admonition:: ECMAScript 5 Changes
+In Flask 0.10 and lower, :func:`~flask.jsonify` did not serialize top-level
-
+arrays to JSON. This was because of a security vulnerability in ECMAScript 4.
-   Starting with ECMAScript 5 the behavior of literals changed.  Now they
+
-   are not constructed with the constructor of ``Array`` and others, but
+ECMAScript 5 closed this vulnerability, so only extremely old browsers are
-   with the builtin constructor of ``Array`` which closes this particular
+still vulnerable. All of these browsers have `other more serious
-   attack vector.
+vulnerabilities
-
+<https://github.com/mitsuhiko/flask/issues/248#issuecomment-59934857>`_, so
-JSON itself is a high-level serialization format, so there is barely
+this behavior was changed and :func:`~flask.jsonify` now supports serializing
-anything that could cause security problems, right?  You can't declare
+arrays.
-recursive structures that could cause problems and the only thing that
-could possibly break are very large responses that can cause some kind of
-denial of service at the receiver's side.
-
-However there is a catch.  Due to how browsers work the CSRF issue comes
-up with JSON unfortunately.  Fortunately there is also a weird part of the
-JavaScript specification that can be used to solve that problem easily and
-Flask is kinda doing that for you by preventing you from doing dangerous
-stuff.  Unfortunately that protection is only there for
-:func:`~flask.jsonify` so you are still at risk when using other ways to
-generate JSON.
-
-So what is the issue and how to avoid it?  The problem are arrays at
-top-level in JSON.  Imagine you send the following data out in a JSON
-request.  Say that's exporting the names and email addresses of all your
-friends for a part of the user interface that is written in JavaScript.
-Not very uncommon:
-
-.. sourcecode:: javascript
-
-    [
-        {"username": "admin",
-         "email": "admin@localhost"}
-    ]
-
-And it is doing that of course only as long as you are logged in and only
-for you.  And it is doing that for all ``GET`` requests to a certain URL,
-say the URL for that request is
-``http://example.com/api/get_friends.json``.
-
-So now what happens if a clever hacker is embedding this to his website
-and social engineers a victim to visiting his site:
-
-.. sourcecode:: html
-
-    <script type=text/javascript>
-    var captured = [];
-    var oldArray = Array;
-    function Array() {
-      var obj = this, id = 0, capture = function(value) {
-        obj.__defineSetter__(id++, capture);
-        if (value)
-          captured.push(value);
-      };
-      capture();
-    }
-    </script>
-    <script type=text/javascript
-      src=http://example.com/api/get_friends.json></script>
-    <script type=text/javascript>
-    Array = oldArray;
-    // now we have all the data in the captured array.
-    </script>
-
-If you know a bit of JavaScript internals you might know that it's
-possible to patch constructors and register callbacks for setters.  An
-attacker can use this (like above) to get all the data you exported in
-your JSON file.  The browser will totally ignore the :mimetype:`application/json`
-mimetype if :mimetype:`text/javascript` is defined as content type in the script
-tag and evaluate that as JavaScript.  Because top-level array elements are
-allowed (albeit useless) and we hooked in our own constructor, after that
-page loaded the data from the JSON response is in the `captured` array.
-
-Because it is a syntax error in JavaScript to have an object literal
-(``{...}``) toplevel an attacker could not just do a request to an
-external URL with the script tag to load up the data.  So what Flask does
-is to only allow objects as toplevel elements when using
-:func:`~flask.jsonify`.  Make sure to do the same when using an ordinary
-JSON generate function.

flask/json.py

@@ -199,10 +199,22 @@ def htmlsafe_dump(obj, fp, **kwargs):
 
 
 def jsonify(*args, **kwargs):
-    """Creates a :class:`~flask.Response` with the JSON representation of
+    """This function wraps :func:`dumps` to add a few enhancements that make
-    the given arguments with an :mimetype:`application/json` mimetype.  The
+    life easier.  It turns the JSON output into a :class:`~flask.Response`
-    arguments to this function are the same as to the :class:`dict`
+    object with the :mimetype:`application/json` mimetype.  For convenience, it
-    constructor.
+    also converts multiple arguments into an array or multiple keyword arguments
+    into a dict.  This means that both ``jsonify(1,2,3)`` and
+    ``jsonify([1,2,3])`` serialize to ``[1,2,3]``.
+
+    For clarity, the JSON serialization behavior has the following differences
+    from :func:`dumps`:
+
+    1. Single argument: Passed straight through to :func:`dumps`.
+    2. Multiple arguments: Converted to an array before being passed to
+    :func:`dumps`.
+    3. Multiple keyword arguments: Converted to a dict before being passed to
+    :func:`dumps`.
+    4. Both args and kwargs: Behavior undefined and will throw an exception.
 
     Example usage::
 
@@ -222,8 +234,10 @@ def jsonify(*args, **kwargs):
             "id": 42
         }
 
-    For security reasons only objects are supported toplevel.  For more
+
-    information about this, have a look at :ref:`json-security`.
+    .. versionchanged:: 1.0
+       Added support for serializing top-level arrays. This introduces a
+       security risk in ancient browsers. See :ref:`json_security` for details.
 
     This function's response will be pretty printed if it was not requested
     with ``X-Requested-With: XMLHttpRequest`` to simplify debugging unless
@@ -242,11 +256,21 @@ def jsonify(*args, **kwargs):
         indent = 2
         separators = (', ', ': ')
 
+    if args and kwargs:
+        raise TypeError(
+            "jsonify() behavior undefined when passed both args and kwargs"
+        )
+    elif len(args) == 1: # single args are passed directly to dumps()
+        data = args[0]
+    elif args: # convert multiple args into an array
+        data = list(args)
+    else: # convert kwargs to a dict
+        data = dict(kwargs)
+
     # Note that we add '\n' to end of response
     # (see https://github.com/mitsuhiko/flask/pull/1262)
     rv = current_app.response_class(
-        (dumps(dict(*args, **kwargs), indent=indent, separators=separators),
+        (dumps(data, indent=indent, separators=separators), '\n'),
-         '\n'),
         mimetype='application/json')
     return rv
 

tests/test_helpers.py

@@ -31,24 +31,6 @@ def has_encoding(name):
 
 class TestJSON(object):
 
-    def test_jsonify_date_types(self):
-        """Test jsonify with datetime.date and datetime.datetime types."""
-
-        test_dates = (
-            datetime.datetime(1973, 3, 11, 6, 30, 45),
-            datetime.date(1975, 1, 5)
-        )
-
-        app = flask.Flask(__name__)
-        c = app.test_client()
-
-        for i, d in enumerate(test_dates):
-            url = '/datetest{0}'.format(i)
-            app.add_url_rule(url, str(i), lambda val=d: flask.jsonify(x=val))
-            rv = c.get(url)
-            assert rv.mimetype == 'application/json'
-            assert flask.json.loads(rv.data)['x'] == http_date(d.timetuple())
-
     def test_post_empty_json_adds_exception_to_response_content_in_debug(self):
         app = flask.Flask(__name__)
         app.config['DEBUG'] = True
@@ -103,8 +85,41 @@ class TestJSON(object):
                      content_type='application/json; charset=iso-8859-15')
         assert resp.data == u'Hällo Wörld'.encode('utf-8')
 
-    def test_jsonify(self):
+    def test_json_as_unicode(self):
-        d = dict(a=23, b=42, c=[1, 2, 3])
+        app = flask.Flask(__name__)
+
+        app.config['JSON_AS_ASCII'] = True
+        with app.app_context():
+            rv = flask.json.dumps(u'\N{SNOWMAN}')
+            assert rv == '"\\u2603"'
+
+        app.config['JSON_AS_ASCII'] = False
+        with app.app_context():
+            rv = flask.json.dumps(u'\N{SNOWMAN}')
+            assert rv == u'"\u2603"'
+
+    def test_jsonify_basic_types(self):
+        """Test jsonify with basic types."""
+        # Should be able to use pytest parametrize on this, but I couldn't
+        # figure out the correct syntax
+        # https://pytest.org/latest/parametrize.html#pytest-mark-parametrize-parametrizing-test-functions
+        test_data = (0, 1, 23, 3.14, 's', "longer string", True, False,)
+        app = flask.Flask(__name__)
+        c = app.test_client()
+        for i, d in enumerate(test_data):
+            url = '/jsonify_basic_types{0}'.format(i)
+            app.add_url_rule(url, str(i), lambda x=d: flask.jsonify(x))
+            rv = c.get(url)
+            assert rv.mimetype == 'application/json'
+            assert flask.json.loads(rv.data) == d
+
+    def test_jsonify_dicts(self):
+        """Test jsonify with dicts and kwargs unpacking."""
+        d = dict(
+            a=0, b=23, c=3.14, d='t', e='Hi', f=True, g=False,
+            h=['test list', 10, False],
+            i={'test':'dict'}
+        )
         app = flask.Flask(__name__)
         @app.route('/kw')
         def return_kwargs():
@@ -118,18 +133,43 @@ class TestJSON(object):
             assert rv.mimetype == 'application/json'
             assert flask.json.loads(rv.data) == d
 
-    def test_json_as_unicode(self):
+    def test_jsonify_arrays(self):
+        """Test jsonify of lists and args unpacking."""
+        l = [
+            0, 42, 3.14, 't', 'hello', True, False,
+            ['test list', 2, False],
+            {'test':'dict'}
+        ]
         app = flask.Flask(__name__)
+        @app.route('/args_unpack')
+        def return_args_unpack():
+            return flask.jsonify(*l)
+        @app.route('/array')
+        def return_array():
+            return flask.jsonify(l)
+        c = app.test_client()
+        for url in '/args_unpack', '/array':
+            rv = c.get(url)
+            assert rv.mimetype == 'application/json'
+            assert flask.json.loads(rv.data) == l
 
-        app.config['JSON_AS_ASCII'] = True
+    def test_jsonify_date_types(self):
-        with app.app_context():
+        """Test jsonify with datetime.date and datetime.datetime types."""
-            rv = flask.json.dumps(u'\N{SNOWMAN}')
-            assert rv == '"\\u2603"'
 
-        app.config['JSON_AS_ASCII'] = False
+        test_dates = (
-        with app.app_context():
+            datetime.datetime(1973, 3, 11, 6, 30, 45),
-            rv = flask.json.dumps(u'\N{SNOWMAN}')
+            datetime.date(1975, 1, 5)
-            assert rv == u'"\u2603"'
+        )
+
+        app = flask.Flask(__name__)
+        c = app.test_client()
+
+        for i, d in enumerate(test_dates):
+            url = '/datetest{0}'.format(i)
+            app.add_url_rule(url, str(i), lambda val=d: flask.jsonify(x=val))
+            rv = c.get(url)
+            assert rv.mimetype == 'application/json'
+            assert flask.json.loads(rv.data)['x'] == http_date(d.timetuple())
 
     def test_json_attr(self):
         app = flask.Flask(__name__)