Policy syncing from API

This document describes how to use OPAL for syncing policy (code & static data) sourced from an API server that exposes tar bundles (rather than from a git repo, which is the default policy source).

Bundle in this context are nothing more than a compressed tarball file archiving your policy files, not to be confused with an OPA Bundle

We have a docker compose example file configured with an api policy source that we will explore in detail later in this guide.

How policy syncing from an API server works

The OPAL server is configured to get its data from API bundle server, extract the bundle, and sync it to the clients. The API server must have a bundle.tar.gz file and be able to serve it to the OPAL server. The OPAL server will always aspire to keep the most up-to-date "state" of the bundle supplied by the bundle server.

Going into greater technical detail - the OPAL Server will:

• Send a request to get the bundle.tar.gz file.

• Extract it to the configured local path.

• Make a git repo from it's content to be able to track changes.

• Upon detecting a new bundle file (tracked by ETag or hashing the file if ETag isn't supported at the API server), the OPAL-server will request and save the new bundle file into its local checkout.

Currently OPAL server supports two ways to detect changes in the tracked repo / bundle server:

• Polling by fixed intervals - checks every OPAL_POLICY_REPO_POLLING_INTERVAL seconds if there's a new bundle file in the API bundle server by running GET <base-url>/bundle.tar.gz periodically.

• Webhook - By issuing an HTTP REST request to OPAL server <opal-server-url>/webhook with your auth access token upon each update bundle file event, you can trigger the OPAL server to fetch a new bundle.

The rest of the policy syncing process is the same as with a git policy source.

Authenticating with Bundle server

You can configure how the OPAL-server will authenticate itself with the bundle server with the following env-var:

Variables	Description	Example
POLICY_BUNDLE_SERVER_TYPE	HTTP (authenticated with bearer token,or nothing), AWS-S3(Authenticated with AWS REST Auth	AWS-S3
POLICY_BUNDLE_SERVER_TOKEN_ID	The Secret Token Id (AKA user id, AKA access-key) sent to the API bundle server.	AKIAIOSFODNN7EXAMPLE
POLICY_BUNDLE_SERVER_TOKEN	The Secret Token (AKA password, AKA secret-key) sent to the API bundle server.	wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
POLICY_BUNDLE_SERVER_AWS_REGION	The AWS Region if using AWS-S3 Defaults to us-east-1	us-east-1

Docker compose example

In this section we show how to configure an API bundle server as OPAL's policy source. We made an example docker-compose.yml file with all the necessary configuration.

Step 1: run docker compose to start the opal server and client

Clone the opal repository and run the example compose file from your local clone:

git clone https://github.com/permitio/opal.git
cd opal
docker-compose -f docker/docker-compose-api-policy-source-example.yml up

The docker-compose.yml we just downloaded (Click here to view its contents) is running 4 containers: Broadcast, OPAL Server, OPAL Client, and API bundle server.

OPAL (and also OPA) are now running on your machine, the following ports are exposed on localhost:

• OPAL Server (port :7002) - the OPAL client (and potentially the cli) can connect to this port.
• OPAL Client (port :7766) - the OPAL client has its own API, but it's irrelevant to this tutorial :)
• OPA (port :8181) - the port of the OPA agent (running in server mode).
  • OPA is being run by OPAL client in its container as a managed process.
• Nginx server that serves a static bundle file (bundle.tar.gz) on port 8000

Step 2: Send some authorization queries to OPA

As mentioned before, the OPA REST API is running on port :8181 - you can issue any requests you'd like to it directly.

Let's explore the current state and send some authorization queries to the agent.

The default policy in the example repo is a simple RBAC policy, we can issue this request to get the user's role assignment and metadata:

curl --request GET 'http://localhost:8181/v1/data/users' --header 'Content-Type: application/json' | python -m json.tool

The expected result is:

{
    "result": {
        "alice": {
            "location": {
                "country": "US",
                "ip": "8.8.8.8"
            },
            "roles": [
                "admin"
            ]
        },

        ...
    }
}

Cool, let's issue an authorization query. In OPA, an authorization query is a query with input.

This query asks whether the user bob can read the finance resource (whose id is id123):

curl -w '\n' --request POST 'http://localhost:8181/v1/data/app/rbac/allow' \
--header 'Content-Type: application/json' \
--data-raw '{"input": {"user": "bob","action": "read","object": "id123","type": "finance"}}'

The expected result is true, meaning the access is granted:

{"result":true}

Step 3: Change the policy, and see it being updated in realtime

Since the example docker-compose-api-policy-source-example.yml makes OPAL track the API bundle server (which serves the files from /docker/docker_files/bundle files). In order to see how a bundle update can affect the policy in realtime, we can run the following commands to trigger a policy update:

cd docker/docker_files/bundle_files
mv bundle.tar.gz{,.bak1}; mv bundle.tar.gz{.bak,}; mv bundle.tar.gz.bak{1,} # this command swaps the two bundle files you have, to trigger a policy change

• You can now run the same query as before (the curl command above) to see that the user's data has changed

Step 4: Publish a data update via the OPAL Server

The default policy in the example repo is a simple RBAC policy with a twist.

A user is granted access if:

• One of his/her role has a permission for the requested action and resource type.
• Only users from the USA can access the resource (location == US).

The reason we added the location policy is we want to show you how pushing an update via opal with a different "user location" can immediately affect access, demonstrating realtime updates needed by most modern applications.

Remember this authorization query?

curl -w '\n' --request POST 'http://localhost:8181/v1/data/app/rbac/allow' \
--header 'Content-Type: application/json' \
--data-raw '{"input": {"user": "bob","action": "read","object": "id123","type": "finance"}}'

Bob is granted access because the initial data.json location is US (link):

{"result":true}

Let's push an update via OPAL and see how poor Bob is denied access.

We can push an update via the opal-client cli. Let's install the cli to a new python virtualenv:

pyenv virtualenv opaldemo
pyenv activate opaldemo
pip install opal-client

Now let's use the cli to push an update to override the user location (we'll come back and explain what we do here in a moment):

opal-client publish-data-update --src-url https://api.country.is/23.54.6.78 -t policy_data --dst-path /users/bob/location

We expect to receive this output from the cli:

Publishing event:
entries=[DataSourceEntry(url='https://api.country.is/23.54.6.78', config={}, topics=['policy_data'], dst_path='/users/bob/location', save_method='PUT')] reason=''
Event Published Successfully

Now let's issue the same authorization query again:

curl -w '\n' --request POST 'http://localhost:8181/v1/data/app/rbac/allow' \
--header 'Content-Type: application/json' \
--data-raw '{"input": {"user": "bob","action": "read","object": "id123","type": "finance"}}'

And..... no dice. Bob is denied access:

{"result":false}

Now, what happened when we published our update with the cli? Let's analyze the components of this update.

OPAL data updates are built to support your specific use case.

• You can specify a topic (in the example: policy_data) to target only specific opal clients (and by extension specific OPA agents). This is only logical if each microservice you have has an OPA sidecar of its own (and different policy/data needs).
• OPAL specifies from where to fetch the data that changed. In this example we used a free and open API (api.country.is) that anyone can access. But it can be your specific API, or a 3rd-party.
• OPAL specifies to where (destination path) in OPA document hierarchy the data should be saved. In this case we override the /users/bob/location document with the fetched data.