Skip to main content

Triggering Data Updates

OPAL allows for other components to notify it (and through it all the OPAL clients , and their next-door policy agents) of data updates, triggering each client [subscribed to the published topic] to fetch the data it needs.

What is this good for?​

Let's try an example - say your application has a billing service, and you want to allow access only to users who have billing enabled (enforced via a policy agent). You now need changes to the state of the billing service to be propagated to each of the enforcement points/agents (and preferably instantly [Users who've paid - don't like to wait 😅 ]). With the OPAL's data-update-triggers feature the billing-service, another service monitoring it, or even a person can trigger updates as they need - knowing OPAL will take it from there to all the points that need it.

First, you need to obtain a data-source identity token (JWT)​

Every service that publishes to OPAL needs a datasource identity token. Obtaining one is easy, but you need access to the corresponding OPAL Server master token.

Obtain a data source token with the cli:

opal-client obtain-token $OPAL_AUTH_MASTER_TOKEN --server-url=$YOUR_SERVERS_ADDRESS --type datasource

If you don't want to use the cli, you can obtain the JWT directly from the deployed OPAL server via its REST API:

curl --request POST 'https://opal.yourdomain.com/token' \
--header 'Authorization: Bearer MY_MASTER_TOKEN' \
--header 'Content-Type: application/json' \
--data-raw '{
  "type": "datasource",
}'

The /token API endpoint can receive more parameters, as documented here.

This example assumes that:

  • You deployed OPAL server to https://opal.yourdomain.com
  • The master token of your deployment is MY_MASTER_TOKEN.
    • In real life, use a cryptographically secure secret. If you followed our tutorials while deploying OPAL, you probably generated one with opal-server generate-secret.

How to trigger updates​

There are a few ways to trigger updates:

Option 1: Trigger a data update with the CLI​

Can be run both from opal-client and opal-server.

Example:

  • With $token being a JWT we generated in step 1.

  • we publish a data-event regarding two topics users and billing pointing clients to http://mybillingserver.com/users to obtain the data they need. we also provide the clients with the credentials they'll need to connect to the server (as HTTP authorization headers)

  • opal-client publish-data-update $token --src-url http://mybillingserver.com/users -t users -t billing --src-config '{"headers":{"authorization":"bearer secret-token"}}'

  • (Yes... We did... We put authorization in your authorization 😜  😅 )

  • See this recording showing the command including getting the JWT for the server with the obtain-token command.

Option 2: Trigger a data update with OPAL Server REST API​

Using PATCH save method​

  • There are two save methods of triggering data update, PUT and PATCH defined in the payload using the save_method field
  • Using PUT basically places(overrides if path exists) data at the specified dst_path
  • When using PATCH, the data field should conform to JSON patch schema according to RFC 6902 and this applies patch operation on the data as specified
  • Let's say the following is the

  • NOTE: OPA does not support move operation of JSON patch, refer discussion for the same and hence OPAL cannot support move operation as well

  • Example: Consider this JSON data from the opal-example-policy-repo
  • Let's say a user is deleted from the system and we would want that user details to be removed from the JSON, let's remove bob from the list, we can use the remove operation of JSON patch to achieve this
  • The following API request will remove bob from the users JSON
 curl -X 'POST' \
  'http://localhost:7002/data/config' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "id": "randomid",
  "entries": [
    {
      "url": "",
      "config": {},
      "topics": [
        "policy_data"
      ],
      "dst_path": "/users",
      "save_method": "PATCH",
      "data": [
        {
          "op": "remove",
          "path": "/bob"
        }
      ]
    }
  ],
  "reason": "user bob is deleted from the system",
  "callback": {
    "callbacks": []
  }
 }'
note

The primariy way meant to do updates with OPAL is with the PUT method; PATCH is better used for light tweaks to the data. While future versions of OPAL might have deeper support for PATCH, it is not currently the case. To build complex data layouts - we recommend building data-updates into keys that do not intersect to build larger data-sets, and when they do intersect we’d recommend updating the entire entry. If you want to use PATCH and have more control over its behavior you can implement a custom data fetcher that will handle the order of writing or dependencies.

Option 3: Write your own - import code from the OPAL's packages​

  • One of the great things about OPAL being written in Python is that you can easily reuse its code. See the code for the DataUpdate model at opal_common/schemas/data.py and use it within your own code to send an update to the server