Pull subscriptions

In pull delivery, your subscriber awarding initiates requests to the Pub/Sub server to retrieve messages.

You can use the pull fashion to perform asynchronous or synchronous message processing. The pull mode runs in a client library that can utilize one of the two service APIs, Pull or StreamingPull. The client library can also exist a high-level or a low-level client library depending on your requirements.

Earlier you lot begin

Before reading this document, ensure that yous are familiar with the following:

  • How Pub/Sub works and the different Pub/Sub terms.

  • The different kinds of subscriptions that Pub/Sub supports and why yous might want to use a pull subscription.

Asynchronous Pull

Using asynchronous pulling provides college throughput in your application, by not requiring your application to block for new messages. Messages tin be received in your application using a long running message listener, and acknowledged one message at a time, as shown in the example below. Java, Python, .Cyberspace, Go, and Ruby clients use the StreamingPull service API to implement the asynchronous client API efficiently.

Non all client libraries support asynchronously pulling messages. To larn about synchronously pulling letters, see Synchronous Pull.

For more data, see the API Reference documentation in your programming language.

C++

Earlier trying this sample, follow the C++ setup instructions in Quickstart: Using Customer Libraries. For more information, see the Pub/Sub C++ API reference documentation.

C#

Before trying this sample, follow the C# setup instructions in Quickstart: Using Customer Libraries. For more information, encounter the Pub/Sub C# API reference documentation.

Become

Before trying this sample, follow the Become setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.

Java

Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more data, run into the Pub/Sub Java API reference documentation.

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, come across the Pub/Sub Node.js API reference documentation.

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more than information, encounter the Pub/Sub Python API reference documentation.

Cherry-red

Before trying this sample, follow the Cherry setup instructions in Quickstart: Using Client Libraries. For more than information, meet the Pub/Sub Ruby API reference documentation.

Processing Custom Attributes

This sample shows how to pull messages asynchronously and retrieve the custom attributes from metadata:

C++

Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.

C#

Before trying this sample, follow the C# setup instructions in Quickstart: Using Customer Libraries. For more than information, run into the Pub/Sub C# API reference documentation.

Go

Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Get API reference documentation.

Java

Before trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more than data, see the Pub/Sub Java API reference documentation.

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more data, see the Pub/Sub Node.js API reference documentation.

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.

Blood-red

Before trying this sample, follow the Ruddy setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Red API reference documentation.

Listening for Errors

This sample shows how to handle errors that arise when subscribing to messages:

C++

Earlier trying this sample, follow the C++ setup instructions in Quickstart: Using Customer Libraries. For more information, encounter the Pub/Sub C++ API reference documentation.

Go

Before trying this sample, follow the Go setup instructions in Quickstart: Using Customer Libraries. For more information, see the Pub/Sub Go API reference documentation.

Java

Earlier trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more than information, encounter the Pub/Sub Node.js API reference documentation.

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more than data, see the Pub/Sub Python API reference documentation.

Ruby

Before trying this sample, follow the Get setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Go API reference documentation.

Message Flow Command

Your subscriber client might process and acknowledge messages more slowly than Pub/Sub sends them to the client. In this case:

  • It's possible that one client could accept a backlog of letters because it doesn't have the capacity to process the volume of incoming messages, only another client on the network does take that chapters. The second client could reduce the subscription'southward backlog, but it doesn't get the chance to do so considering the starting time client maintains a lease on the messages that it receives. This reduces the overall rate of processing considering messages become stuck on the first client.

  • Because the client library repeatedly extends the acknowledgement deadline for backlogged messages, those letters continue to swallow retentivity, CPU, and bandwidth resource. Equally such, the subscriber client might run out of resources (such every bit memory). This can adversely impact the throughput and latency of processing messages.

To mitigate the issues higher up, use the flow control features of the subscriber to command the rate at which the subscriber receives messages. These flow control features are illustrated in the following samples:

C++

Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C++ API reference documentation.

C#

Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more data, see the Pub/Sub C# API reference documentation.

Become

Before trying this sample, follow the Go setup instructions in Quickstart: Using Customer Libraries. For more information, see the Pub/Sub Go API reference documentation.

Java

Earlier trying this sample, follow the Java setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, run into the Pub/Sub Node.js API reference documentation.

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Python API reference documentation.

Ruby

Before trying this sample, follow the Blood-red setup instructions in Quickstart: Using Client Libraries. For more information, encounter the Pub/Sub Ruby API reference documentation.

More than by and large, the demand for menses control indicates that messages are being published at a higher rate than they are existence consumed. If this is a persistent state, rather than a transient spike in message volume, consider increasing the number of subscriber client instances.

Concurrency Control

Support for concurrency depends on your programming language. For language implementations that support parallel threads, such equally Java and Get, the customer libraries make a default choice for the number of threads. This option may not exist optimal for your awarding. For instance, if you notice that your subscriber application is not keeping up with the incoming message volume but is non CPU-bound, you should increment the thread count. For CPU-intensive message processing operations, reducing the number of threads might be appropriate.

The following samples illustrate how to control concurrency in a subscriber:

Support for concurrency depends on your programming language. Refer to the API Reference documentation for more data.

Exactly in one case delivery

With exactly once commitment, no redelivery occurs once the message has been successfully acknowledged. This means that the acknowledgement future for a received bulletin must return successfully.

The following sample illustrates how to receive letters from a subscription with exactly once delivery enabled:

Using schema

These samples testify how to procedure messages when subscribing to messages on topics configured with schema. To learn more, run into Creating and managing schemas.

C++

Before trying this sample, follow the C++ setup instructions in Quickstart: Using Client Libraries. For more information, run into the Pub/Sub C++ API reference documentation.

Avro Proto

C#

Earlier trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more than data, see the Pub/Sub C# API reference documentation.

Avro Proto

Go

Before trying this sample, follow the Go setup instructions in Quickstart: Using Client Libraries. For more information, encounter the Pub/Sub Get API reference documentation.

Avro Proto

Java

Before trying this sample, follow the Go setup instructions in Quickstart: Using Customer Libraries. For more information, encounter the Pub/Sub Go API reference documentation.

Avro Protocol Buffer

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, meet the Pub/Sub Node.js API reference documentation.

Avro Protocol Buffer

PHP

Before trying this sample, follow the PHP setup instructions in Quickstart: Using Client Libraries. For more than information, come across the Pub/Sub PHP API reference documentation.

Avro Protocol Buffer

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, run into the Pub/Sub Python API reference documentation.

Avro Protocol Buffer

Ruby

Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Cherry API reference documentation.

Avro Protocol Buffer

StreamingPull

The Pub/Sub service has two APIs for retrieving letters:

  • Pull
  • StreamingPull

Where possible, the Cloud Client libraries use StreamingPull for maximum throughput and everyman latency. Although y'all might never use the StreamingPull API directly, it is important to empathise some crucial properties of StreamingPull and how it differs from the more traditional Pull method.

The Pull method relies on a request/response model:

  1. The client sends a request to the server for messages.
  2. If the user is out of throughput quota, the server returns a RESOURCE_EXHAUSTED error.
  3. The server replies with zippo or more messages and closes the connexion.

The StreamingPull service API relies on a persistent bidirectional connexion to receive multiple letters as they become bachelor:

  1. The customer sends a request to the server to establish a connection.
  2. If the user is out of connections quota, the server returns a RESOURCE_EXHAUSTED error.
  3. The server continuously sends messages to the connected client.
  4. If/when the user runs out of throughput quota, the stream is suspended, merely the connection is not cleaved. When in that location is sufficient throughput quota bachelor again, the connection is resumed.
  5. The connectedness is somewhen closed either by the customer or the server.

You provide a callback to the subscriber and the subscriber asynchronously runs the callback for each message. If a subscriber receives messages with the same ordering key, the client libraries sequentially run the callback. The Pub/Sub service delivers these letters to the same subscriber on a best-effort basis.

StreamingPull has a 100% error rate (this is to exist expected)

StreamingPull streams always shut with a not-OK condition. Note that, unlike in regular RPCs, the status here is simply an indication that the stream has been broken, not that requests are declining. Therefore, while the StreamingPull API may accept a seemingly surprising 100% error rate, this is past design.

Diagnosing StreamingPull errors

Because StreamingPull streams ever shut with an fault, it isn't helpful to examine stream termination metrics while diagnosing errors. Rather, focus on the StreamingPull response metric ( subscription/streaming_pull_response_count ). Await for these errors:

  • FAILED_PRECONDITION errors tin can occur in these cases:
    • Pub/Sub attempts to decrypt a message with a disabled Cloud KMS key.
    • Subscriptions can be temporarily suspended if there are letters in the subscription excess that are encrypted with a disabled Cloud KMS primal.
  • UNAVAILABLE errors

StreamingPull: Dealing with large backlogs of small letters

The gRPC StreamingPull stack is optimized for high throughput and therefore buffers messages. This can take some consequences if you are attempting to procedure large backlogs of small messages (rather than a steady stream of new messages). Under these conditions, you may come across messages delivered multiple times and they may not be load balanced effectively across clients.

The buffer between the Pub/Sub service and the client library user space is roughly 10MB. To sympathize the impact of this buffer on client library beliefs, consider this case:

  • There is a backlog of 10,000 1KB letters on a subscription.
  • Each message takes 1 second to procedure sequentially, by a single-threaded client case.
  • The first customer case to establish a StreamingPull connexion to the service for that subscription will fill up its buffer with all 10,000 messages.
  • Information technology takes 10,000 seconds (almost 3 hours) to process the buffer.
  • In that fourth dimension, some of the buffered messages exceed their acknowledgement deadline and are re-sent to the same client, resulting in duplicates.
  • When multiple customer instances are running, the messages stuck in the 1 client's buffer volition non be available to whatever client instances.

This situation volition not occur if y'all use flow command for StreamingPull: the service never has the entire 10MB of messages at a time then is able to effectively load remainder letters across multiple subscribers.

To address this state of affairs, either use a push button subscription or the Pull API, currently available in some of the Deject Client Libraries (come across the Synchronous Pull section) and all API Customer libraries. To learn more than, run into the Client Libraries documentation.

Synchronous Pull

In that location are cases when the asynchronous Pull is non a perfect fit for your application. For case, the awarding logic might rely on a polling design to retrieve messages or require a precise cap on a number of messages retrieved past the customer at any given time. To back up such applications, the service supports a synchronous Pull method.

Here is some sample code to pull and acknowledge a fixed number of messages:

C#

Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub C# API reference documentation.

Go

Before trying this sample, follow the Become setup instructions in Quickstart: Using Client Libraries. For more than information, see the Pub/Sub Go API reference documentation.

Java

Before trying this sample, follow the Coffee setup instructions in Quickstart: Using Customer Libraries. For more than information, meet the Pub/Sub Java API reference documentation.

Node.js

Before trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more than information, see the Pub/Sub Node.js API reference documentation.

PHP

Earlier trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, run across the Pub/Sub Node.js API reference documentation.

Protocol

Asking:

Postal service https://pubsub.googleapis.com/v1/projects/myproject/subscriptions/mysubscription:pull 

                        {   "returnImmediately": "fake",   "maxMessages": "1" }

Response:

200 OK 

                        {   "receivedMessages": [{     "ackId": "dQNNHlAbEGEIBERNK0EPKVgUWQYyODM2LwgRHFEZDDsLRk1SK...",     "message": {       "data": "SGVsbG8gQ2xvdWQgUHViL1N1YiEgSGVyZSBpcyBteSBtZXNzYWdlIQ==",       "messageId": "19917247034"     }   }] }

Request:

POST https://pubsub.googleapis.com/v1/projects/myproject/subscriptions/mysubscription:admit 

                        {   "ackIds": [     "dQNNHlAbEGEIBERNK0EPKVgUWQYyODM2LwgRHFEZDDsLRk1SK..."   ] }

Python

Earlier trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more data, see the Pub/Sub Python API reference documentation.

Reddish

Before trying this sample, follow the Ruby setup instructions in Quickstart: Using Client Libraries. For more information, run into the Pub/Sub Cherry-red API reference documentation.

Pub/Sub delivers a listing of messages. If the list has multiple messages, Pub/Sub orders the messages with the same ordering key.

Synchronous Pull with Lease Management

An individual message's processing may exceed the preconfigured acknowledgement borderline, also known as the charter. A good utilize example for synchronous pull with charter management is when y'all are processing hundreds of messages in parallel, and each bulletin takes a long time to procedure. To avoid redelivery on expired messages, the client libraries provide a way to reset their acknowledgement deadlines (except for the Go client library, which automatically modifies the acknowledgement deadlines for polled messages), equally shown by the samples beneath:

C#

Before trying this sample, follow the C# setup instructions in Quickstart: Using Client Libraries. For more than information, encounter the Pub/Sub C# API reference documentation.

Java

Before trying this sample, follow the Coffee setup instructions in Quickstart: Using Client Libraries. For more information, see the Pub/Sub Java API reference documentation.

Node.js

Earlier trying this sample, follow the Node.js setup instructions in Quickstart: Using Client Libraries. For more information, run across the Pub/Sub Node.js API reference documentation.

Python

Before trying this sample, follow the Python setup instructions in Quickstart: Using Client Libraries. For more information, run across the Pub/Sub Python API reference documentation.

Ruby

Before trying this sample, follow the Blood-red setup instructions in Quickstart: Using Client Libraries. For more data, see the Pub/Sub Cerise API reference documentation.

Scaling

You lot may need to implement a scaling mechanism for your subscriber application to keep up with bulletin volume. How to do this depends on your environment, just it will mostly exist based on backlog metrics offered through the Google Cloud'south operations suite monitoring service. For details on how to do this for Compute Engine, see Scaling based on Cloud Monitoring Metrics.

Go to the Pub/Sub department of the GCP Metrics List page to learn which metrics tin exist monitored programmatically.

Finally, equally with all distributed services, expect to occasionally retry every request.

Dealing with duplicates and forcing retries

When you do non acknowledge a message before its acknowledgement borderline has expired, Pub/Sub resends the bulletin. As a consequence, Pub/Sub can transport indistinguishable messages. Use Cloud Monitoring to monitor admit operations with the expired response lawmaking to find this condition. To get this data, select the subscription/expired_ack_deadlines_count metric.

Use Cloud Monitoring to to search for expired message acknowledgement deadlines

To reduce the duplication charge per unit, extend the message deadline.

  • Client libraries handle borderline extension automatically, simply you should note that there are default limits on the maximum extension deadline that tin exist configured.
  • If y'all are building your own client library, use the modifyAckDeadline method to extend the acknowledgement deadline.

Alternately, to force Pub/Sub to retry a message, set modifyAckDeadline to 0.