📖Pagination v2

Introduction

When building API integrations, you may encounter a scenario where the server does not provide all the results in a single list. Instead, it divides the results into several pages. This strategy is called pagination, and integrating it into your application can be tedious and repetitive. With Saloon, you can install this pagination plugin to reduce the boilerplate code and iterate through every result across every page in one loop. It's like magic! Saloon has three paginators out of the box to support the most common types of pagination:

• Paged pagination - where you have to specify the next page as a query parameter (?page=1, ?page=2 and so on)

• Limit/Offset pagination - where you must specify two query parameters - a limit and an offset.

• Cursor pagination - where each page will return a "cursor" (a string) that you use to query the next page.

Saloon's paginators are custom PHP iterators, meaning they can be used in for-loops. They are also memory efficient, so they only keep one page in memory at a time, this means you can use Saloon's paginators to iterate through thousands of pages and millions of results without running out of memory.

Additionally, Saloon's paginators have a few really useful methods to traverse through the results, like mapping them into Laravel's famous "Collection" class or simply an array of results. Here's how the pagination looks in your application:

<?php

$spotifyConnector = new SpotifyConnector;

$likedSongs = $spotifyConnector->paginate(new GetLikedSongsRequest);

foreach($likedSongs->items() as $likedSong) {
    // e.g. Luke Combs - When It Rains It Pours 🎵
}

Saloon's paginators are versatile and can work with any API format, including JSON, XML, and even multipart responses.

Getting Started

Saloon's paginators are provided through a plugin. First, install the following package with Composer.

composer require saloonphp/pagination-plugin "^1.0"

Making your requests pageable

Once the pagination plugin is installed, you must implement an interface on the requests that use pagination. This interface enables the paginator to decide whether or not to try pagination. It's important to do this as it avoids developer mistakes and potentially causing infinite loops.

All you need to do is add the Paginatable interface to your requests.

<?php

use Saloon\Http\Request;
use Saloon\PaginationPlugin\Contracts\Paginatable;

class GetLikedSongsRequest extends Request implements Paginatable
{
    // ...
}

After that, you need to add the HasPagination interface to your connector. This will require you to define a paginate method which we will implement next.

<?php

use Saloon\Http\Connector;
use Saloon\PaginationPlugin\Paginator;
use Saloon\PaginationPlugin\Contracts\HasPagination;

class SpotifyConnector extends Connector implements HasPagination
{
    // ...
    
    public function paginate(Request $request): Paginator
    {
        //
    }
}

Take care when importing the pagination plugin classes when using Saloon v2. You'll want to make sure you are using the Saloon\PaginationPlugin namespace.

Now you need to choose a paginator that corresponds to the pagination type used by the third-party API. This documentation is divided into three sections based on Saloon's three pre-installed paginators.

Paged Pagination

Limit/Offset Pagination

Cursor Pagination

Building Your Own Paginators

Sometimes an API may implement their own kind of pagination and Saloon might not have a pagination supported out of the box for it. You can create your own paginators really easily. Follow the link below for building your own paginators.

Custom Pagination

Using The Paginator

After you have configured your paginator, it's time to start using it! There are various ways to use the paginator, so we'll cover them in this part of the guide. First, let's take a look at how we instantiate our paginator. When we instantiate the paginator - Saloon won't make any requests. The requests are only made when we start iterating over the paginator.

<?php

$spotifyConnector = new SpotifyConnector;

$paginator = $spotifyConnector->paginate(new GetLikedSongsRequest);

As you can see, instantiating the paginator is as simple as running the method and passing in the paginatable request. We are now ready to start using the paginator.

Iterating Over Responses

The simplest way to start using the paginator is to use it inside of a foreach loop. This will return each page as a Saloon Response for you to access. Saloon will only keep one response in memory at a time so you won't have to worry about running out of memory.

<?php

$spotifyConnector = new SpotifyConnector;

$paginator = $spotifyConnector->paginate(new GetLikedSongsRequest);

foreach($paginator as $response) {
    $status = $response->status();
    $data = $response->body();
}

Iterating Over Items

One of the more exciting methods Saloon provides is the items method. This cool method will give you each item across multiple pages. With this, you can handle the individual items. This uses the getPageItems method which you defined earlier in your paginator.

<?php

$spotifyConnector = new SpotifyConnector;

$paginator = $spotifyConnector->paginate(new GetLikedSongsRequest);

foreach($paginator->items() as $song) {
    $songName = $song['name'];
    $songArtists = $song['artists'];
}

Using Laravel Collections (LazyCollection)

Another really cool method Saloon provides is the collect method. If you are using Laravel with Saloon, this method will work out of the box, but if you aren't using Laravel - you can add the support for this method by installing the "collections" library via Composer.

composer require illuminate/collections

This method will return a LazyCollection class. This class extends Laravel's base Collection class but works with custom iterators and keeps memory consumption low. The LazyCollection is really powerful when it comes to processing results from a paginated API. You can use the collection to filter, map, sort and much more all within just one iteration!

<?php

$spotifyConnector = new SpotifyConnector;

$paginator = $spotifyConnector->paginate(new GetLikedSongsRequest);
$collection = $paginator->collect();

$results = $collection
    ->filter(function (array $song) {
        return count($song['artists']) > 1;
    })
    ->map(function (array $song) {
        return $song['name'];
    })
    ->sort();
    
foreach($results as $song) {
    // $song: "When It Rains It Pours"
}

Even though in this example, we are doing multiple operations (filter, map, sort) the transformation will only happen when we iterate through the LazyCollection making it very performant and easy on system memory.

By default, Saloon will iterate through each result within each API page, but if you would rather have a collection that iterates through Response classes, you may do this too. Just set the throughItems argument to false.

<?php

$spotifyConnector = new SpotifyConnector;

$paginator = $spotifyConnector->paginate(new GetLikedSongsRequest);

$collection = $paginator->collect(throughItems: false);

To see a full list of available methods on the LazyCollection you can click here to read the Laravel Documentation on Collections.

Advanced Features

Saloon's paginators are a fantastic way of iterating through many page-separated results quickly and easily without writing boilerplate code in your application. So far we've covered the basics of pagination, but there's still plenty you can do. Within this section of the guide, we'll go through some more advanced features.

Casting Items Into DTOs

Saloon has a great way of casting responses into DTOs, but what if instead of getting the raw array or string item for each item in the paginator you wanted to resolve your DTO? You can simply use your dto or dtoOrFail method inside of the getPageItems method of your paginator. By using this method, you already have configured your DTO building logic on your connector. If you're not sure how to do this, click here.

Paginator Config

class SpotifyConnector extends Connector implements HasPagination
{
    public function paginate(Request $request): PagedPaginator
    {
        return new class(connector: $this, request: $request) extends PagedPaginator
        {
            // ...
            
            protected function getPageItems(Response $response, Request $request): array
            {
                return $response->dto();
            }
        }
    }
}

Request

<?php

class GetLikedSongsRequest extends Request
{
    // {...}
    
    public function createDtoFromResponse(Response $response): array
    {
        return array_map(function (array $song) {
            return Song::fromArray($song);
        }, $response->json('items'));
    }
}

The createDtoFromResponse method on your request/connector must return an array.

You can also implement the DTO conversion logic inside of the getPageItems method like below.

protected function getPageItems(Response $response, Request $request): array
{
    return array_map(function (array $song) {
        return Song::fromArray($song);
    }, $response->json('items'));
}

Asynchronous Pagination / Pooling

You may prefer to send your requests asynchronously or use a request pool for better performance. To get started with this, you must implement a new method on your paginator called getTotalPages. This method is invoked after the first request and is used to determine how many additional requests Saloon needs to send.

Let's start with defining our getTotalPages method. Some APIs may not return this information, and unfortunately, if they do not return this information then you may not be able to do asynchronous pagination.

<?php

class SpotifyConnector extends Connector implements HasPagination
{
    public function paginate(Request $request): PagedPaginator
    {
        return new class(connector: $this, request: $request) extends PagedPaginator
        {
            // ...
            
            protected function getTotalPages(Response $response): int
            {
                return $response->json('total_results');
            }
        }
    }
}

Next, instantiate your paginator and use the async method to enable asynchronous pagination. Now when you iterate through results, you will get a Promise instance instead of a Saloon Response.

<?php

$spotifyConnector = new SpotifyConnector;

$paginator = $spotifyConnector->paginate(new GetLikedSongsRequest);

$paginator->async();

foreach($paginator as $promise) {
    // $promise->then()
}

You can also use a paginator pool. If you are unfamiliar with how asynchronous request pools work, check out this section of the documentation first.

<?php

$spotifyConnector = new SpotifyConnector;

$paginator = $spotifyConnector->paginate(new GetLikedSongsRequest);

$pool = $paginator->pool(concurrency: 10);

$pool->withResponseHandler(function (Response $response) { 
    // Handle each response

    $status = $response->status();
    $data = $response->body(); 
});

$pool->wait();

Custom Per-Request Pagination

Sometimes you may wish to use a different paginator on your request than your connector. For example, an API might not use the same pagination across all its endpoints. To get started with this - first, implement the HasRequestPagination interface on your request. This interface will expect you to define a paginate method on your request.

<?php

use Saloon\PaginationPlugin\Contracts\HasRequestPagination;

class GetLikedSongsRequest extends Request implements HasRequestPagination
{
    // {...}
    
    public function paginate(Connector $connector): Paginator;
    {
        //
    }
}

This method works just like the paginate method on your connector, so follow the steps above to configure your paginator. The main difference is that this method expects a connector to be passed into the constructor and not a request.

<?php

use Saloon\PaginationPlugin\Contracts\HasRequestPagination;

class GetLikedSongsRequest extends Request implements HasRequestPagination
{
    // {...}
    
    public function paginate(Connector $connector): Paginator;
    {
        return new class(connector: $connector, request: $this) extends PagedPaginator
        {
            protected function isLastPage(Response $response): bool
            {
                return is_null($response->json('next_page_url'));
            }
            
            protected function getPageItems(Response $response, Request $request): array
            {
                return $response->json('items');
            }
        }
    }
}

Next, you can pass the connector into your request to instantiate the paginator.

<?php

$spotifyConnector = new SpotifyConnector;
$request = new GetLikedSongsRequest;

$paginator = $request->paginate($spotifyConnector);

Counting results

You may wish to count the pages of the paginator. You can use the count() PHP method to get the total number of pages from a paginator.

<?php

$spotifyConnector = new SpotifyConnector;

$paginator = $spotifyConnector->paginate(new GetLikedSongsRequest);

$pages = count($paginator);

You should use the count() method and not iterator_count() on the paginator because iterator_count() is not supported and you may get inaccurate results.

Handling Infinite Loops

While building your paginators, sometimes, you might run into a situation when the paginator is attempting to get the same page over and over again, causing an infinite loop. This is usually caused by mis-configuring the isLastPage method. To try to mitigate infinite loops, the paginator has a built-in safety feature that will throw an exception if the last five responses have exactly the same body.

Saloon will throw a PaginationException with a message like:

Potential infinite loop detected! The last 5 requests have had exactly the same body.

If you would like to disable this check, you can set the detectInfiniteLoop property on your paginator to false.

public function paginate(Connector $connector): Paginator;
{
    return new class(connector: $connector, request: $this) extends PagedPaginator
    {
        protected bool $detectInfiniteLoop = false;
    }
}

Detecting infinite loops with asynchronous pagination is not supported.