Merge pull request #2395 from tomchristie/pagination-api

First pass at 3.1 pagination API

6 files changed, 192 insertions, 200 deletions

diff --git a/docs/api-guide/pagination.md b/docs/api-guide/pagination.md
index 83429292..9fbeb22a 100644
--- a/docs/api-guide/pagination.md
+++ b/docs/api-guide/pagination.md
@@ -6,148 +6,101 @@ source: pagination.py
 >
 > — [Django documentation][cite]
 
-REST framework includes a `PaginationSerializer` class that makes it easy to return paginated data in a way that can then be rendered to arbitrary media types.
+REST framework includes support for customizable pagination styles. This allows you to modify how large result sets are split into individual pages of data.
 
-## Paginating basic data
+The pagination API can support either:
 
-Let's start by taking a look at an example from the Django documentation.
+* Pagination links that are provided as part of the content of the response.
+* Pagination links that are included in response headers, such as `Content-Range` or `Link`.
 
-    from django.core.paginator import Paginator
+The built-in styles currently all use links included as part of the content of the response. This style is more accessible when using the browsable API.
 
-    objects = ['john', 'paul', 'george', 'ringo']
-    paginator = Paginator(objects, 2)
-    page = paginator.page(1)
-    page.object_list
-    # ['john', 'paul']
+Pagination is only performed automatically if you're using the generic views or viewsets. If you're using a regular `APIView`, you'll need to call into the pagination API yourself to ensure you return a paginated response. See the source code for the `mixins.ListMixin` and `generics.GenericAPIView` classes for an example.
 
-At this point we've got a page object.  If we wanted to return this page object as a JSON response, we'd need to provide the client with context such as next and previous links, so that it would be able to page through the remaining results.
+## Setting the pagination style
 
-    from rest_framework.pagination import PaginationSerializer
+The default pagination style may be set globally, using the `DEFAULT_PAGINATION_CLASS` settings key. For example, to use the built-in limit/offset pagination, you would do:
 
-    serializer = PaginationSerializer(instance=page)
-    serializer.data
-    # {'count': 4, 'next': '?page=2', 'previous': None, 'results': [u'john', u'paul']}
-
-The `context` argument of the `PaginationSerializer` class may optionally include the request.  If the request is included in the context then the next and previous links returned by the serializer will use absolute URLs instead of relative URLs.
-
-    request = RequestFactory().get('/foobar')
-    serializer = PaginationSerializer(instance=page, context={'request': request})
-    serializer.data
-    # {'count': 4, 'next': 'http://testserver/foobar?page=2', 'previous': None, 'results': [u'john', u'paul']}
-
-We could now return that data in a `Response` object, and it would be rendered into the correct media type.
-
-## Paginating QuerySets
-
-Our first example worked because we were using primitive objects.  If we wanted to paginate a queryset or other complex data, we'd need to specify a serializer to use to serialize the result set itself.
-
-We can do this using the `object_serializer_class` attribute on the inner `Meta` class of the pagination serializer.  For example.
-
-    class UserSerializer(serializers.ModelSerializer):
-        """
-        Serializes user querysets.
-        """
-        class Meta:
-            model = User
-            fields = ('username', 'email')
+    REST_FRAMEWORK = {
+        'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination'
+    }
 
-    class PaginatedUserSerializer(pagination.PaginationSerializer):
-        """
-        Serializes page objects of user querysets.
-        """
-        class Meta:
-            object_serializer_class = UserSerializer
+You can also set the pagination class on an individual view by using the `pagination_class` attribute. Typically you'll want to use the same pagination style throughout your API, although you might want to vary individual aspects of the pagination, such as default or maximum page size, on a per-view basis.
 
-We could now use our pagination serializer in a view like this.
+## Modifying the pagination style
 
-    @api_view('GET')
-    def user_list(request):
-        queryset = User.objects.all()
-        paginator = Paginator(queryset, 20)
+If you want to modify particular aspects of the pagination style, you'll want to override one of the pagination classes, and set the attributes that you want to change.
 
-        page = request.QUERY_PARAMS.get('page')
-        try:
-            users = paginator.page(page)
-        except PageNotAnInteger:
-            # If page is not an integer, deliver first page.
-            users = paginator.page(1)
-        except EmptyPage:
-            # If page is out of range (e.g. 9999),
-            # deliver last page of results.
-            users = paginator.page(paginator.num_pages)
+    class LargeResultsSetPagination(PageNumberPagination):
+        paginate_by = 1000
+        paginate_by_param = 'page_size'
+        max_paginate_by = 10000
 
-        serializer_context = {'request': request}
-        serializer = PaginatedUserSerializer(users,
-                                             context=serializer_context)
-        return Response(serializer.data)
+    class StandardResultsSetPagination(PageNumberPagination):
+        paginate_by = 100
+        paginate_by_param = 'page_size'
+        max_paginate_by = 1000
 
-## Pagination in the generic views
+You can then apply your new style to a view using the `.pagination_class` attribute:
 
-The generic class based views `ListAPIView` and `ListCreateAPIView` provide pagination of the returned querysets by default.  You can customise this behaviour by altering the pagination style, by modifying the default number of results, by allowing clients to override the page size using a query parameter, or by turning pagination off completely.
+    class BillingRecordsView(generics.ListAPIView):
+        queryset = Billing.objects.all()
+        serializer = BillingRecordsSerializer
+        pagination_class = LargeResultsSetPagination
 
-The default pagination style may be set globally, using the `DEFAULT_PAGINATION_SERIALIZER_CLASS`, `PAGINATE_BY`, `PAGINATE_BY_PARAM`, and `MAX_PAGINATE_BY` settings.  For example.
+Or apply the style globally, using the `DEFAULT_PAGINATION_CLASS` settings key. For example:
 
     REST_FRAMEWORK = {
-        'PAGINATE_BY': 10,                 # Default to 10
-        'PAGINATE_BY_PARAM': 'page_size',  # Allow client to override, using `?page_size=xxx`.
-        'MAX_PAGINATE_BY': 100             # Maximum limit allowed when using `?page_size=xxx`.
-    }
-
-You can also set the pagination style on a per-view basis, using the `ListAPIView` generic class-based view.
+        'DEFAULT_PAGINATION_CLASS': 'apps.core.pagination.StandardResultsSetPagination'
    }
 
-    class PaginatedListView(ListAPIView):
-        queryset = ExampleModel.objects.all()
-        serializer_class = ExampleModelSerializer
-        paginate_by = 10
-        paginate_by_param = 'page_size'
-        max_paginate_by = 100
+# API Reference
 
-Note that using a `paginate_by` value of `None` will turn off pagination for the view.
-Note if you use the `PAGINATE_BY_PARAM` settings, you also have to set the `paginate_by_param` attribute in your view to `None` in order to turn off pagination for those requests that contain the `paginate_by_param` parameter.
+## PageNumberPagination
 
-For more complex requirements such as serialization that differs depending on the requested media type you can override the `.get_paginate_by()` and `.get_pagination_serializer_class()` methods.
+## LimitOffsetPagination
 
 ---
 
-# Custom pagination serializers
+# Custom pagination styles
+
+To create a custom pagination serializer class you should subclass `pagination.BasePagination` and override the `paginate_queryset(self, queryset, request, view)` and `get_paginated_response(self, data)` methods:
 
-To create a custom pagination serializer class you should override `pagination.BasePaginationSerializer` and set the fields that you want the serializer to return.
+* The `paginate_queryset` method is passed the initial queryset and should return an iterable object that contains only the data in the requested page.
+* The `get_paginated_response` method is passed the serialized page data and should return a `Response` instance.
 
-You can also override the name used for the object list field, by setting the `results_field` attribute, which defaults to `'results'`.
+Note that the `paginate_queryset` method may set state on the pagination instance, that may later be used by the `get_paginated_response` method.
 
 ## Example
 
-For example, to nest a pair of links labelled 'prev' and 'next', and set the name for the results field to 'objects', you might use something like this.
+Let's modify the built-in `PageNumberPagination` style, so that instead of include the pagination links in the body of the response, we'll instead include a `Link` header, in a [similar style to the GitHub API][github-link-pagination].
 
-    from rest_framework import pagination
-    from rest_framework import serializers
+    class LinkHeaderPagination(PageNumberPagination)
+        def get_paginated_response(self, data):
+            next_url = self.get_next_link()
            previous_url = self.get_previous_link()
 
-    class LinksSerializer(serializers.Serializer):
-        next = pagination.NextPageField(source='*')
-        prev = pagination.PreviousPageField(source='*')
+            if next_url is not None and previous_url is not None:
+                link = '<{next_url}; rel="next">, <{previous_url}; rel="prev">'
+            elif next_url is not None:
+                link = '<{next_url}; rel="next">'
+            elif prev_url is not None:
+                link = '<{previous_url}; rel="prev">'
+            else:
+                link = ''
 
-    class CustomPaginationSerializer(pagination.BasePaginationSerializer):
-        links = LinksSerializer(source='*')  # Takes the page object as the source
-        total_results = serializers.ReadOnlyField(source='paginator.count')
+            link = link.format(next_url=next_url, previous_url=previous_url)
+            headers = {'Link': link} if link else {}
 
-        results_field = 'objects'
+            return Response(data, headers=headers)
 
-## Using your custom pagination serializer
+## Using your custom pagination class
 
-To have your custom pagination serializer be used by default, use the `DEFAULT_PAGINATION_SERIALIZER_CLASS` setting:
+To have your custom pagination class be used by default, use the `DEFAULT_PAGINATION_CLASS` setting:
 
     REST_FRAMEWORK = {
-        'DEFAULT_PAGINATION_SERIALIZER_CLASS':
-            'example_app.pagination.CustomPaginationSerializer',
+        'DEFAULT_PAGINATION_CLASS':
+            'my_project.apps.core.pagination.LinkHeaderPagination',
     }
 
-Alternatively, to set your custom pagination serializer on a per-view basis, use the `pagination_serializer_class` attribute on a generic class based view:
-
-    class PaginatedListView(generics.ListAPIView):
-        model = ExampleModel
-        pagination_serializer_class = CustomPaginationSerializer
-        paginate_by = 10
-
 # Third party packages
 
 The following third party packages are also available.
@@ -157,5 +110,6 @@ The following third party packages are also available.
 The [`DRF-extensions` package][drf-extensions] includes a [`PaginateByMaxMixin` mixin class][paginate-by-max-mixin] that allows your API clients to specify `?page_size=max` to obtain the maximum allowed page size.
 
 [cite]: https://docs.djangoproject.com/en/dev/topics/pagination/
+[github-link-pagination]: https://developer.github.com/guides/traversing-with-pagination/
 [drf-extensions]: http://chibisov.github.io/drf-extensions/docs/
 [paginate-by-max-mixin]: http://chibisov.github.io/drf-extensions/docs/#paginatebymaxmixin
diff --git a/docs/index.md b/docs/index.md
index 544204c6..16376985 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -305,6 +305,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 [settings]: api-guide/settings.md
 
 [documenting-your-api]: topics/documenting-your-api.md
+[internationalization]: topics/documenting-your-api.md
 [ajax-csrf-cors]: topics/ajax-csrf-cors.md
 [browser-enhancements]: topics/browser-enhancements.md
 [browsableapi]: topics/browsable-api.md
diff --git a/docs/topics/3.1-announcement.md b/docs/topics/3.1-announcement.md
new file mode 100644
index 00000000..a0ad9829
--- /dev/null
+++ b/docs/topics/3.1-announcement.md
@@ -0,0 +1,7 @@
+# Versioning
+
+# Pagination
+
+# Internationalization
+
+# ModelSerializer API
diff --git a/docs/topics/internationalisation.md b/docs/topics/internationalisation.md
deleted file mode 100644
index 2a476c86..00000000
--- a/docs/topics/internationalisation.md
+++ /dev/null
@@ -1,95 +0,0 @@
-# Internationalisation
-REST framework ships with translatable error messages.  You can make these appear in your language enabling [Django's standard translation mechanisms][django-translation] and by translating the messages into your language.
-
-## How to translate REST Framework errors
-
-REST framework translations are managed online using [Transifex.com][transifex]. To get started, checkout the guide in the [CONTRIBUTING.md guide][contributing].
-
-Sometimes you may want to use REST Framework in a language which has not been translated yet on Transifex. If that is the case then you should translate the error messages locally.
-
-#### How to translate REST Framework error messages locally:
-
-This guide assumes you are already familiar with how to translate a Django app.  If you're not, start by reading [Django's translation docs][django-translation].
-
-1. Make a new folder where you want to store the translated errors. Add this 
-path to your [`LOCALE_PATHS`][django-locale-paths] setting. 
-
-  ---
-
-  **Note:** For the rest of 
-this document we will assume the path you created was 
-`/home/www/project/conf/locale/`, and that you have updated your `settings.py` to include the setting:
-
-  ```
-  LOCALE_PATHS = (
-      '/home/www/project/conf/locale/',
-  )
-  ```
-
-  ---
-
-2. Now create a subfolder for the language you want to translate. The folder should be named using [locale 
-name][django-locale-name] notation.  E.g. `de`, `pt_BR`, `es_AR`, etc.
-
-  ```
-  mkdir /home/www/project/conf/locale/pt_BR/LC_MESSAGES
-  ```
-
-3. Now copy the base translations file from the REST framework source code 
-into your translations folder
-
-  ```
-  cp /home/user/.virtualenvs/myproject/lib/python2.7/site-packages/rest_framework/locale/en_US/LC_MESSAGES/django.po
-  /home/www/project/conf/locale/pt_BR/LC_MESSAGES
-  ```
-  
-  This should create the file 
-  `/home/www/project/conf/locale/pt_BR/LC_MESSAGES/django.po`
-  
-  ---
-
-  **Note:** To find out where `rest_framework` is installed, run 
-
-  ```
-  python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
-  ```
-
-  ---
-  
-  
-4. Edit `/home/www/project/conf/locale/pt_BR/LC_MESSAGES/django.po` and 
-translate all the error messages.
-
-5. Run `manage.py compilemessages -l pt_BR` to make the translations 
-available for Django to use. You should see a message
-
-    ```
-    processing file django.po in /home/www/project/conf/locale/pt_BR/LC_MESSAGES
-    ```
-
-6. Restart your server.
-
-
-
-## How Django chooses which language to use
-REST framework will use the same preferences to select which language to 
-display as Django does.  You can find more info in the [Django docs on discovering language preferences][django-language-preference].  For reference, these are
-
-1. First, it looks for the language prefix in the requested URL
-2. Failing that, it looks for the `LANGUAGE_SESSION_KEY` key in the current user’s session.
-3. Failing that, it looks for a cookie
-4. Failing that, it looks at the `Accept-Language` HTTP header.
-5. Failing that, it uses the global `LANGUAGE_CODE` setting.
-
----
-
-**Note:** You'll need to include the `django.middleware.locale.LocaleMiddleware` to enable any of the per-request language preferences.
-
----
-
-
-[django-translation]: https://docs.djangoproject.com/en/1.7/topics/i18n/translation
-[django-language-preference]: https://docs.djangoproject.com/en/1.7/topics/i18n/translation/#how-django-discovers-language-preference
-[django-locale-paths]: https://docs.djangoproject.com/en/1.7/ref/settings/#std:setting-LOCALE_PATHS
-[django-locale-name]: https://docs.djangoproject.com/en/1.7/topics/i18n/#term-locale-name
-[contributing]: ../../CONTRIBUTING.md
diff --git a/docs/topics/internationalization.md b/docs/topics/internationalization.md
new file mode 100644
index 00000000..fdde6c43
--- /dev/null
+++ b/docs/topics/internationalization.md
@@ -0,0 +1,72 @@
+# Internationalization
+
+> Supporting internationalization is not optional. It must be a core feature.
+>
+> &mdash; [Jannis Leidel, speaking at Django Under the Hood, 2015][cite].
+
+REST framework ships with translatable error messages. You can make these appear in your language enabling [Django's standard translation mechanisms][django-translation].
+
+Doing so will allow you to:
+
+* Select a language other than English as the default, using the standard `LANGUAGE_CODE` Django setting.
+* Allow clients to choose a language themselves, using the `LocaleMiddleware` included with Django. A typical usage for API clients would be to include an `Accept-Language` request header.
+
+Note that the translations only apply to the error strings themselves. The format of error messages, and the keys of field names will remain the same. An example `400 Bad Request` response body might look like this:
+
+    {"detail": {"username": ["Esse campo deve ser unico."]}}
+
+If you want to use different string for parts of the response such as `detail` and `non_field_errors` then you can modify this behavior by using a [custom exception handler][custom-exception-handler].
+
+## Adding new translations
+
+REST framework translations are managed online using [Transifex][transifex-project]. You can use the Transifex service to add new translation languages. The maintenance team will then ensure that these translation strings are included in the REST framework package.
+
+Sometimes you may need to add translation strings to your project locally. You may need to do this if:
+
+* You want to use REST Framework in a language which has not been translated yet on Transifex.
+* Your project includes custom error messages, which are not part of REST framework's default translation strings.
+
+#### Translating a new language locally
+
+This guide assumes you are already familiar with how to translate a Django app.  If you're not, start by reading [Django's translation docs][django-translation].
+
+If you're translating a new language you'll need to translate the existing REST framework error messages:
+
+1. Make a new folder where you want to store the internationalization resources. Add this path to your [`LOCALE_PATHS`][django-locale-paths] setting.
+
+2. Now create a subfolder for the language you want to translate. The folder should be named using [locale name][django-locale-name] notation. For example: `de`, `pt_BR`, `es_AR`.
+
+3. Now copy the [base translations file][django-po-source] from the REST framework source code into your translations folder.
+
+4. Edit the `django.po` file you've just copied, translating all the error messages.
+
+5. Run `manage.py compilemessages -l pt_BR` to make the translations 
+available for Django to use. You should see a message like `processing file django.po in <...>/locale/pt_BR/LC_MESSAGES`.
+
+6. Restart your development server to see the changes take effect.
+
+If you're only translating custom error messages that exist inside your project codebase you don't need to copy the REST framework source `django.po` file into a `LOCALE_PATHS` folder, and can instead simply run Django's standard `makemessages` process.
+
+## How the language is determined
+
+If you want to allow per-request language preferences you'll need to include `django.middleware.locale.LocaleMiddleware` in your `MIDDLEWARE_CLASSES` setting.
+
+You can find more information on how the language preference is determined in the [Django documentation][django-language-preference]. For reference, the method is:
+
+1. First, it looks for the language prefix in the requested URL.
+2. Failing that, it looks for the `LANGUAGE_SESSION_KEY` key in the current user’s session.
+3. Failing that, it looks for a cookie.
+4. Failing that, it looks at the `Accept-Language` HTTP header.
+5. Failing that, it uses the global `LANGUAGE_CODE` setting.
+
+For API clients the most appropriate of these will typically be to use the `Accept-Language` header; Sessions and cookies will not be available unless using session authentication, and generally better practice to prefer an `Accept-Language` header for API clients rather than using language URL prefixes.
+
+[cite]: http://youtu.be/Wa0VfS2q94Y
+[django-translation]: https://docs.djangoproject.com/en/1.7/topics/i18n/translation
+[custom-exception-handler]: ../api-guide/exceptions.md#custom-exception-handling

+[transifex-project]: https://www.transifex.com/projects/p/django-rest-framework/
+[django-po-source]: https://raw.githubusercontent.com/tomchristie/django-rest-framework/master/rest_framework/locale/en_US/LC_MESSAGES/django.po 
+[django-language-preference]: https://docs.djangoproject.com/en/1.7/topics/i18n/translation/#how-django-discovers-language-preference
+[django-locale-paths]: https://docs.djangoproject.com/en/1.7/ref/settings/#std:setting-LOCALE_PATHS
+[django-locale-name]: https://docs.djangoproject.com/en/1.7/topics/i18n/#term-locale-name
+[contributing]: ../../CONTRIBUTING.md
diff --git a/docs/topics/project-management.md b/docs/topics/project-management.md
index f581cabd..7f705196 100644
--- a/docs/topics/project-management.md
+++ b/docs/topics/project-management.md
@@ -63,10 +63,11 @@ The following template should be used for the description of the issue, and serv
 
 Team members have the following responsibilities.
 
-* Add triage labels and milestones to tickets.
 * Close invalid or resolved tickets.
+* Add triage labels and milestones to tickets.
 * Merge finalized pull requests.
 * Build and deploy the documentation, using `mkdocs gh-deploy`.
+* Build and update the included translation packs.
 
 Further notes for maintainers:
 
@@ -112,6 +113,55 @@ When pushing the release to PyPI ensure that your environment has been installed
 
 ---
 
+## Translations
+
+The maintenance team are responsible for managing the translation packs include in REST framework. Translating the source strings into multiple languages is managed through the [transifex service][transifex-project].
+
+### Managing Transifex
+
+The [official Transifex client][transifex-client] is used to upload and download translations to Transifex. The client is installed using pip:
+
+    pip install transifex-client
+
+To use it you'll need a login to Transifex which has a password, and you'll need to have administrative access to the Transifex project. You'll need to create a `~/.transifexrc` file which contains your credentials.
+
+    [https://www.transifex.com]
+    username = ***
+    token = ***
+    password = ***
+    hostname = https://www.transifex.com
+
+### Upload new source files
+
+When any user visible strings are changed, they should be uploaded to Transifex so that the translators can start to translate them. To do this, just run:
+
+    # 1. Update the source django.po file, which is the US English version.
+    cd rest_framework
+    django-admin.py makemessages -l en_US
+    # 2. Push the source django.po file to Transifex.
+    cd ..
+    tx push -s
+
+When pushing source files, Transifex will update the source strings of a resource to match those from the new source file.
+
+Here's how differences between the old and new source files will be handled:
+
+* New strings will be added.
+* Modified strings will be added as well.
+* Strings which do not exist in the new source file will be removed from the database, along with their translations. If that source strings gets re-added later then [Transifex Translation Memory][translation-memory] will automatically include the translation string.
+
+### Download translations
+
+When a translator has finished translating their work needs to be downloaded from Transifex into the REST framework repository. To do this, run:
+
+    # 3. Pull the translated django.po files from Transifex.
+    tx pull -a
+    cd rest_framework
+    # 4. Compile the binary .mo files for all supported languages.
+    django-admin.py compilemessages
+
+---
+
 ## Project ownership
 
 The PyPI package is owned by `@tomchristie`. As a backup `@j4mie` also has ownership of the package.
@@ -129,6 +179,9 @@ The following issues still need to be addressed:
 
 [bus-factor]: http://en.wikipedia.org/wiki/Bus_factor
 [un-triaged]: https://github.com/tomchristie/django-rest-framework/issues?q=is%3Aopen+no%3Alabel
+[transifex-project]: https://www.transifex.com/projects/p/django-rest-framework/
+[transifex-client]: https://pypi.python.org/pypi/transifex-client
+[translation-memory]: http://docs.transifex.com/guides/tm#let-tm-automatically-populate-translations
 [github-org]: https://github.com/tomchristie/django-rest-framework/issues/2162
 [sandbox]: http://restframework.herokuapp.com/
 [mailing-list]: https://groups.google.com/forum/#!forum/django-rest-framework