Iterators#

Iterators for paging through API responses.

These iterators simplify the process of paging through API responses where the response is a list of results with a nextPageToken.

To make an iterator work, you’ll need to provide a way to convert a JSON item returned from the API into the object of your choice (via item_to_value). You also may need to specify a custom items_key so that a given response (containing a page of results) can be parsed into an iterable page of the actual objects you want. You then can use this to get all the results from a resource:

>>> def item_to_value(iterator, item):
...     my_item = MyItemClass(iterator.client, other_arg=True)
...     my_item._set_properties(item)
...     return my_item
...
>>> iterator = Iterator(..., items_key='blocks',
...                     item_to_value=item_to_value)
>>> list(iterator)  # Convert to a list (consumes all values).

Or you can walk your way through items and call off the search early if you find what you’re looking for (resulting in possibly fewer requests):

>>> for my_item in Iterator(...):
...     print(my_item.name)
...     if not my_item.is_valid:
...         break

At any point, you may check the number of items consumed by referencing the num_results property of the iterator:

>>> my_iterator = Iterator(...)
>>> for my_item in my_iterator:
...     if my_iterator.num_results >= 10:
...         break

When iterating, not every new item will send a request to the server. To iterate based on each page of items (where a page corresponds to a request):

>>> iterator = Iterator(...)
>>> for page in iterator.pages:
...     print('=' * 20)
...     print('    Page number: %d' % (iterator.page_number,))
...     print('  Items in page: %d' % (page.num_items,))
...     print('     First item: %r' % (next(page),))
...     print('Items remaining: %d' % (page.remaining,))
...     print('Next page token: %s' % (iterator.next_page_token,))
====================
    Page number: 1
  Items in page: 1
     First item: <MyItemClass at 0x7f1d3cccf690>
Items remaining: 0
Next page token: eav1OzQB0OM8rLdGXOEsyQWSG
====================
    Page number: 2
  Items in page: 19
     First item: <MyItemClass at 0x7f1d3cccffd0>
Items remaining: 18
Next page token: None

To consume an entire page:

>>> list(page)
[
    <MyItemClass at 0x7fd64a098ad0>,
    <MyItemClass at 0x7fd64a098ed0>,
    <MyItemClass at 0x7fd64a098e90>,
]
google.cloud.iterator.DEFAULT_ITEMS_KEY = 'items'#

The dictionary key used to retrieve items from each response.

class google.cloud.iterator.GAXIterator(client, page_iter, item_to_value, max_results=None)[source]#

Bases: google.cloud.iterator.Iterator

A generic class for iterating through Cloud gRPC APIs list responses.

Parameters:
  • client (Client) – The client used to identify the application.
  • page_iter (PageIterator) – A GAX page iterator to be wrapped and conform to the Iterator surface.
  • item_to_value (callable) – Callable to convert an item from a protobuf into the native object. Assumed signature takes an Iterator and a single item from the API response as a protobuf.
  • max_results (int) – (Optional) The maximum number of results to fetch.
pages#

Iterator of pages in the response.

Return type:GeneratorType
Returns:A generator of Page instances.
Raises:ValueError – If the iterator has already been started.
class google.cloud.iterator.HTTPIterator(client, path, item_to_value, items_key='items', page_token=None, max_results=None, extra_params=None, page_start=<function _do_nothing_page_start>)[source]#

Bases: google.cloud.iterator.Iterator

A generic class for iterating through Cloud JSON APIs list responses.

Parameters:
  • client (Client) – The client used to identify the application.
  • path (str) – The path to query for the list of items.
  • item_to_value (callable) – Callable to convert an item from JSON into the native object. Assumed signature takes an Iterator and a dictionary holding a single item.
  • items_key (str) – (Optional) The key used to grab retrieved items from an API response. Defaults to DEFAULT_ITEMS_KEY.
  • page_token (str) – (Optional) A token identifying a page in a result set.
  • max_results (int) – (Optional) The maximum number of results to fetch.
  • extra_params (dict) – (Optional) Extra query string parameters for the API call.
  • page_start (callable) – (Optional) Callable to provide any special behavior after a new page has been created. Assumed signature takes the Iterator that started the page, the Page that was started and the dictionary containing the page response.
pages#

Iterator of pages in the response.

Return type:GeneratorType
Returns:A generator of Page instances.
Raises:ValueError – If the iterator has already been started.
class google.cloud.iterator.Iterator(client, item_to_value, page_token=None, max_results=None)[source]#

Bases: object

A generic class for iterating through API list responses.

Parameters:
  • client (Client) – The client used to identify the application.
  • item_to_value (callable) – Callable to convert an item from the type in the raw API response into the native object. Assumed signature takes an Iterator and a raw API response with a single item.
  • page_token (str) – (Optional) A token identifying a page in a result set.
  • max_results (int) – (Optional) The maximum number of results to fetch.
pages#

Iterator of pages in the response.

Return type:GeneratorType
Returns:A generator of Page instances.
Raises:ValueError – If the iterator has already been started.
class google.cloud.iterator.Page(parent, items, item_to_value)[source]#

Bases: object

Single page of results in an iterator.

Parameters:
  • parent (Iterator) – The iterator that owns the current page.
  • items (iterable) – An iterable (that also defines __len__) of items from a raw API response.
  • item_to_value (callable) – Callable to convert an item from the type in the raw API response into the native object. Assumed signature takes an Iterator and a raw API response with a single item.
next()[source]#

Get the next value in the page.

num_items#

Total items in the page.

Return type:int
Returns:The number of items in this page.
remaining#

Remaining items in the page.

Return type:int
Returns:The number of items remaining in this page.