HTTP Feeds

Asynchronous event streaming and data replication with plain HTTP APIs.

HTTP feeds is a minimal specification for polling events over HTTP:

HTTP feeds can be used to decouple systems asynchronously without message brokers, such as Kafka or RabbitMQ.

Example

GET /inventory HTTP/1.1
Host: https://example.http-feeds.org

HTTP/1.1 200 OK
Content-Type: application/cloudevents-batch+json

[{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "1c6b8c6e-d8d0-4a91-b51c-1f56bd04c758",
  "time" : "2021-01-01T00:00:01Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:01Z",
    "quantity": 5
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "292042fb-ab04-4653-af90-19a24032bffe",
  "time" : "2021-12-01T00:00:15Z",
  "subject" : "9521234512349",
  "data" : {
    "sku": "9521234512349",
    "updated": "2022-01-01T00:00:12Z",
    "quantity": 0
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6",
  "time" : "2021-01-01T00:00:22Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:21Z",
    "quantity": 4
  }
}]

Client calls again with the last processed event id.

GET /inventory?lastEventId=fa3e2a22-398c-4d02-ad08-9415e43178e6 HTTP/1.1
Host: https://example.http-feeds.org

HTTP/1.1 200 OK
Content-Type: application/cloudevents-batch+json

[]

An empty array signals the end of the feed.

Polling

A client can continue polling in an infinite loop to subscribe to a feed.

Simple Polling

A client calls the endpoint with the last known event id as lastEventId query parameter in an endless loop. If the response is an empty array, the client reached the end of the stream and waits some time to make another call to get events that occured in the meantime.

Pseudocode:

endpoint = "https://example.http-feeds.org/inventory"
lastEventId = null

while true:
  try:
    response = GET endpoint + "?lastEventId=" + lastEventId 
    for event in response:
      process event
      lastEventId = event.id
    if response is empty:
      wait N seconds 
  except:
    wait N seconds

Client must persist the id of the last processed event as lastEventId for further fetches.

The client’s event processing must be idempotent (at-least-once delivery semantic). The id may be used for idempotency checks.

Long Polling

The server may also support long polling for lower latency. The client adds a timeout query parameter to specify the max period of milliseconds to wait for an response.

Client Pseudocode:

endpoint = "https://example.http-feeds.org/inventory"
lastEventId = null
timeout = 5000 // 5000 milliseconds is a good timeout period for long polling

while true:
  try:
    response = GET endpoint + "?lastEventId=" + lastEventId + "&timeout=" + timeout
    for event in response:
      process event
      lastEventId = event.id
    // no client wait step within the loop
  except:
    // Delay the next request only in case of a server error
    wait N seconds

If there are no newer events available, the server keeps the connection open until new events arrive or the defined period timed out. The server then sends the response (with the new events or an empty array) and the client can immediately perform another call.

The latency can be improved, as the server can react to new events efficiently by implementing an internal event notification, change data capture triggers, or performing a high-frequency polling to the database.

The cost of long polling is that the server needs to handle more open connections concurrently. This may become an issue with more than 10K connections.

Event ID

The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events.

Events may also get deleted from the feed, e.g. through Compaction and Deletion. The server must still respect the original position and send only newer events, even when an event with the lastEventId was deleted.

One way to achieve this is to use an time-ordered UUIDv6 as event ID (see IETF draft and Java-Library). This a viable option, especially if only one server appends new events to the feed, but might be a problem with multiple servers when the clocks are not in-sync.

An alternative is to use a database sequence that is used as part of the event.id and interpreted when querying the database for the next batch. Example: The event.id 0000001000001::5f8de8ff-30d8-4fab-8f5a-c32f326d6f26 contains a database sequence 0000001000001 and a random UUID.

Event Feeds

HTTP feeds can be used to provide an API for publishing immutable domain events to other systems.

It is quite common that one http feed endpoint includes different event types that belong to the same bounded context.

Aggregate Feeds

HTTP feeds can be used to provide an API for data collections of mutable objects (aka aggregates, master data) to other systems.

An aggregate is identified through its subject. An aggregate feed must contain every aggregate at least once. Every created aggregate and each update leads to an appended feed entry with the full current state.

Feed consumers can subscribe an aggregate feed to perform near real-time data synchronization to build local read models and to trigger actions when new or updated data is received. A feed consumer has a consistent state when reaching the end of the feed.

The server should implement Compaction and may implement Deletion based on business requirements.

Compaction

Each aggregate update leads to an additional entry in the feed. It is good practice to keep the feed small to enable a quick synchronization of new clients. When feed items include the full current state of the resource, older feed items for the same aggregate may be outdated.

Clients that read the feed from the beginning will therefore occur an inconsistent state for a short time. To mitigate, entries may be deleted from the feed when another entry was appended to the feed with the same subject.

Example: There is an update for subject 9521234567899.

HTTP/1.1 200 OK
Content-Type: application/cloudevents-batch+json

[{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "1c6b8c6e-d8d0-4a91-b51c-1f56bd04c758",
  "time" : "2021-01-01T00:00:01Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:01Z",
    "quantity": 5
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "292042fb-ab04-4653-af90-19a24032bffe",
  "time" : "2021-12-01T00:00:15Z",
  "subject" : "9521234512349",
  "data" : {
    "sku": "9521234512349",
    "updated": "2022-01-01T00:00:12Z",
    "quantity": 0
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6",
  "time" : "2021-01-01T00:00:22Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:21Z",
    "quantity": 4
  }
}]

After a compaction run, the first entry is gone:

HTTP/1.1 200 OK
Content-Type: application/cloudevents-batch+json

[{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "292042fb-ab04-4653-af90-19a24032bffe",
  "time" : "2021-12-01T00:00:15Z",
  "subject" : "9521234512349",
  "data" : {
    "sku": "9521234512349",
    "updated": "2022-01-01T00:00:12Z",
    "quantity": 0
  }
},{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "fa3e2a22-398c-4d02-ad08-9415e43178e6",
  "time" : "2021-01-01T00:00:22Z",
  "subject" : "9521234567899",
  "data" : {
    "sku": "9521234567899",
    "updated": "2022-01-01T00:00:21Z",
    "quantity": 4
  }
}]

Deletion

Some aggregates need to be deleted, e. g. by regulatory requirements.

Aggregate feeds use a method field with value DELETE to signal the deletion of an subject to consumers that built a local read model before.

When aggregate was deleted, the server must append a DELETE entry with the subject to delete and no data content.

{
  "specversion" : "1.0",
  "type" : "org.http-feeds.example.inventory",
  "source" : "https://example.http-feeds.org/inventory",
  "id" : "06b13630-e4c3-4d85-a669-ce66fc4daa75",
  "time" : "2021-12-31T00:00:01Z",
  "subject" : "9521234567899",
  "method": "DELETE"
}

Clients must delete this aggregate or otherwise handle the removal.

The server should start a compaction run afterwards to delete previous entries for the same aggregate.

Data Model

An HTTP feed endpoint must support these query parameters:

Query Parameter Type Mandatory Description
lastEventId String Optional The last event id that was processed. Used to scroll through further events. May be missing or null to start from the beginning of the feed.
timeout Number Optional A timeout is set, when long-polling should be used and is supported by the server. Max waiting time in milliseconds for long-polling, after which the server must send a response. A typical value is 5000.

The response body contains an array of events that comply with the CloudEvents Specification in the application/cloudevents-batch+json format.

Field Type Event Feed Aggregate Feed Description
specversion String Mandatory Mandatory The currently supported CloudEvents specification version.
id String Mandatory Mandatory A unique value (such as a UUID) for this event. It can be used to implement deduplication/idempotency handling in downstream systems. It is used as the lastEventId in subsequent calls.
type String Mandatory Mandatory The type of the event. May be used to specify and deserialize the payload. A feed may contain different event types. It SHOULD be prefixed with a reverse-DNS name.
source String Mandatory Mandatory The source system that created the event. Should be a URI of the system.
time String Mandatory Mandatory The event addition timestamp. ISO 8601 UTC date and time format.
subject String n/a Mandatory Key to identify the business object. It doesn’t have to be unique within the feed. This should represent a business key such as an order number or sku. Used for compaction and deletion, if implemented.
method String n/a Optional The HTTP equivalent method type that the feed item performs on the subject. PUT indicates that the subject was created or updated. DELETE indicates that the subject was deleted. Defaults to PUT.
datacontenttype String Optional Optional Defaults to application/json.
data Object Mandatory Optional The payload of the item. Defaults to JSON. May be missing, e.g. when the method was DELETE.

Further metadata may be added, e.g. for traceability.

Authentication

HTTP feeds may be protected with HTTP authentication.

The most common authentication schemes are Basic and Bearer.

The server may filter feed items based on the principal. When filtering is applied, caching may be unfeasible.

Caching

Servers may set appropriate response headers, such as Cache-Control: public, max-age=31536000, when a batch is full and will not be modified anymore.

Libraries and Examples

About

This site is maintained by Jochen Christ. Jochen works as tech lead at INNOQ and is also author of WhichJDK.com, Remote Mob Programming, and Data Mesh Architecture.

HTTP feeds has evolved from rest-feeds.

Contributions are highly appreciated, e.g. libraries or examples in different languages or frameworks help a lot.

Found an error or something is missing? Please raise an issue or create a pull request.

This specification is published under CC BY 4.0.