Memory Leak in OPAL Client During Reconnection

[goncalo-silva-prt]

Describe the bug
OPAL client experiences a progressive memory leak that causes memory utilization to increase by approximately 10% every time the client reconnects to the OPAL server. The memory never decreases after the reconnection, leading to unbounded memory growth over time.
The memory spike occurs immediately after "Performing data configuration, reason: Initial load" log message.

To Reproduce
Our opal-client and server are deployed in ECS (18 client tasks, 2 server tasks). From time to time (happens periodically, but inconsistently - can be every hour or every 10 hours):

1. opal-server closes the connection with opal-client
2. opal-client reconnects to opal-server and downloads the initial data (log: Performing data configuration, reason: Initial load)
3. opal-client memory increases ~10%

OPAL-server env variables

{
"name": "OPAL_BROADCAST_URI",
"value": "kafka://..."
},
{
"name": "KAFKA_SECURITY_PROTOCOL",
"value": "SASL_SSL"
},
{
"name": "KAFKA_SASL_MECHANISM",
"value": "SCRAM-SHA-512"
},
{
"name": "KAFKA_PLAIN_USERNAME",
"valueFrom": "{FROM-SECRETS}"
},
{
"name": "KAFKA_PLAIN_PASSWORD",
"valueFrom": "{FROM-SECRETS}"
},
{
"name": "UVICORN_NUM_WORKERS",
"value": "4"
},
{
"name": "OPAL_REPO_WATCHER_ENABLED",
"value": "false"
},
{
"name": "OPAL_DATA_CONFIG_SOURCES",
"value": "{\"config\":{\"entries\":[{\"url\":\"http://identity.local/api/users\",\"config\":{\"fetcher\":\"HttpFetchProvider\",\"topics\":[\"policy_data\"],\"dst_path\":\"users\"}]}}"
},
{
"name": "OPAL_LOG_SERIALIZE",
"value": "TRUE"
},
{
"name": "DD_TRACE_ENABLED",
"value": "false"
}

OPAL-client env variables

{
    "name": "OPAL_SERVER_URL",
    "value": "http://opal-server.local"
},
{
    "name": "OPAL_INLINE_OPA_LOG_FORMAT",
    "value": "http"
},
{
    "name": "OPAL_POLICY_UPDATER_ENABLED",
    "value": "false"
},
{
    "name": "OPAL_FETCH_PROVIDER_MODULES",
    "valueFrom": "opal_common.fetcher.providers"
}

OPAL Server Logs

Nov 14 06:01:45.059 i-057b4106e35082453 opal-server Authenticated as ZTX5ILH3WW57CEUM via SCRAM-SHA-512
Nov 14 06:01:36.931 i-057b4106e35082453 opal-server Authenticated as ZTX5ILH3WW57CEUM via SCRAM-SHA-512
Nov 14 06:01:00.061 i-057b4106e35082453 opal-server 10.0.3.92:27014 - "GET /data/config HTTP/1.1" 200
Nov 14 06:01:00.061 i-057b4106e35082453 opal-server Serving source configuration
Nov 14 06:01:00.058 i-057b4106e35082453 opal-server 10.0.3.92:27006 - "GET /data/config HTTP/1.1" 200
Nov 14 06:01:00.057 i-057b4106e35082453 opal-server Serving source configuration
Nov 14 06:01:00.057 i-057b4106e35082453 opal-server 10.0.1.69:63620 - "GET /data/config HTTP/1.1" 200
Nov 14 06:01:00.056 i-057b4106e35082453 opal-server Serving source configuration
Nov 14 06:01:00.052 i-008606528e3b4be48 opal-server 10.0.1.69:37956 - "GET /data/config HTTP/1.1" 200
Nov 14 06:01:00.052 i-008606528e3b4be48 opal-server Serving source configuration
Nov 14 06:01:00.049 i-057b4106e35082453 opal-server connection open
Nov 14 06:01:00.049 i-057b4106e35082453 opal-server Client connected
Nov 14 06:01:00.046 i-057b4106e35082453 opal-server connection open
Nov 14 06:01:00.046 i-057b4106e35082453 opal-server Client connected
Nov 14 06:01:00.043 i-008606528e3b4be48 opal-server connection open
Nov 14 06:01:00.043 i-008606528e3b4be48 opal-server Client connected
Nov 14 06:01:00.038 i-008606528e3b4be48 opal-server connection open
Nov 14 06:01:00.038 i-008606528e3b4be48 opal-server Client connected
Nov 14 06:01:00.037 i-008606528e3b4be48 opal-server connection closed
Nov 14 06:01:00.037 i-008606528e3b4be48 opal-server connection closed
Nov 14 06:01:00.036 i-008606528e3b4be48 opal-server Client disconnected - 3144 :: 926f6450d8174bddb87aebf89d6d8a7a
Nov 14 06:01:00.036 i-008606528e3b4be48 opal-server Client disconnected - 7312 :: edc10954ee744d5283d48e658206068b
Nov 14 06:01:00.036 i-057b4106e35082453 opal-server connection closed
Nov 14 06:01:00.036 i-057b4106e35082453 opal-server Client disconnected - 52992 :: 94f5c533af3d4025ade3385cc1baa214
Nov 14 06:01:00.030 i-008606528e3b4be48 opal-server connection closed
Nov 14 06:01:00.030 i-008606528e3b4be48 opal-server Client disconnected - 41702 :: 3e534fc898604c4cb4a98ef4a0650238
Nov 14 06:00:05.294 i-008606528e3b4be48 opal-server Authenticated as ZTX5ILH3WW57CEUM via SCRAM-SHA-512
Nov 14 05:56:46.373 i-008606528e3b4be48 opal-server Authenticated as ZTX5ILH3WW57CEUM via SCRAM-SHA-512
Nov 14 06:00:05.294 i-008606528e3b4be48 opal-server Authenticated as ZTX5ILH3WW57CEUM via SCRAM-SHA-512
Nov 14 05:56:46.373 i-008606528e3b4be48 opal-server Authenticated as ZTX5ILH3WW57CEUM via SCRAM-SHA-512
Nov 14 05:56:45.381 i-057b4106e35082453 opal-server Authenticated as ZTX5ILH3WW57CEUM via SCRAM-SHA-512

OPAL Client Logs

Nov 14 06:01:04.860 i-008606528e3b4be48 opal-client Sent response. PUT /v1/data/users -> 204
Nov 14 06:01:04.684 i-008606528e3b4be48 opal-client Received request. PUT /v1/data/users
Nov 14 06:01:04.662 i-0dd6e5eac040e388e opal-client Sent response. PUT /v1/data/users -> 204
Nov 14 06:01:04.532 i-0c1eaa111e87f8a30 opal-client Sent response. PUT /v1/data/users -> 204
Nov 14 06:01:04.465 i-0dd6e5eac040e388e opal-client Received request. PUT /v1/data/users
Nov 14 06:01:04.460 i-057b4106e35082453 opal-client Sent response. PUT /v1/data/users -> 204
Nov 14 06:01:04.332 i-0c1eaa111e87f8a30 opal-client Received request. PUT /v1/data/users
Nov 14 06:01:04.251 i-057b4106e35082453 opal-client Received request. PUT /v1/data/users
Nov 14 06:01:02.751 i-008606528e3b4be48 opal-client Saving fetched data to policy-store: source url='http://identity.local/api/users', destination path='/users'
Nov 14 06:01:02.486 i-0dd6e5eac040e388e opal-client Saving fetched data to policy-store: source url='http://identity.local/api/users', destination path='/users'
Nov 14 06:01:02.282 i-0c1eaa111e87f8a30 opal-client Saving fetched data to policy-store: source url='http://identity.local/api/users', destination path='/users'
Nov 14 06:01:02.093 i-057b4106e35082453 opal-client Saving fetched data to policy-store: source url='http://identity.local/api/users', destination path='/users'
Nov 14 06:01:00.064 i-008606528e3b4be48 opal-client Connected to PubSub server ws://opal-server.local/ws
Nov 14 06:01:00.063 i-008606528e3b4be48 opal-client Fetching data from url: http://identity.local/api/users
Nov 14 06:01:00.063 i-008606528e3b4be48 opal-client Triggering data update with id: 07ee779b5da542ce87f8f02bdbfe6b7a
Nov 14 06:01:00.061 i-057b4106e35082453 opal-client Connected to PubSub server ws://opal-server.local/ws
Nov 14 06:01:00.060 i-057b4106e35082453 opal-client Fetching data from url: http://identity.local/api/users
Nov 14 06:01:00.059 i-057b4106e35082453 opal-client Triggering data update with id: 3112c32649194c4c947986a97fbd04f4
Nov 14 06:01:00.059 i-0c1eaa111e87f8a30 opal-client Connected to PubSub server ws://opal-server.local/ws
Nov 14 06:01:00.058 i-0c1eaa111e87f8a30 opal-client Fetching data from url: http://identity.local/api/users
Nov 14 06:01:00.058 i-0c1eaa111e87f8a30 opal-client Triggering data update with id: 7957cf2509ae4819a47535a8a974fd27
Nov 14 06:01:00.057 i-008606528e3b4be48 opal-client Getting data-sources configuration from 'http://opal-server.local/data/config'
Nov 14 06:01:00.057 i-008606528e3b4be48 opal-client Performing data configuration, reason: Initial load
Nov 14 06:01:00.057 i-008606528e3b4be48 opal-client Connected to server
Nov 14 06:01:00.055 i-0dd6e5eac040e388e opal-client Connected to PubSub server ws://opal-server.local/ws
Nov 14 06:01:00.054 i-0dd6e5eac040e388e opal-client Fetching data from url: http://identity.local/api/users
Nov 14 06:01:00.054 i-0dd6e5eac040e388e opal-client Triggering data update with id: 2625087308014ae498f49ab05c26d2d9
Nov 14 06:01:00.053 i-057b4106e35082453 opal-client Getting data-sources configuration from 'http://opal-server.local/data/config'
Nov 14 06:01:00.053 i-057b4106e35082453 opal-client Performing data configuration, reason: Initial load
Nov 14 06:01:00.052 i-057b4106e35082453 opal-client Connected to server
Nov 14 06:01:00.052 i-0c1eaa111e87f8a30 opal-client Getting data-sources configuration from 'http://opal-server.local/data/config'
Nov 14 06:01:00.052 i-0c1eaa111e87f8a30 opal-client Performing data configuration, reason: Initial load
Nov 14 06:01:00.052 i-0c1eaa111e87f8a30 opal-client Connected to server
Nov 14 06:01:00.047 i-0dd6e5eac040e388e opal-client Getting data-sources configuration from 'http://opal-server.local/data/config'
Nov 14 06:01:00.047 i-0dd6e5eac040e388e opal-client Performing data configuration, reason: Initial load
Nov 14 06:01:00.047 i-0dd6e5eac040e388e opal-client Connected to server
Nov 14 06:01:00.037 i-057b4106e35082453 opal-client Trying server - ws://opal-server.local/ws
Nov 14 06:01:00.037 i-057b4106e35082453 opal-client Trying to connect to Pub/Sub server - ws://opal-server.local/ws
Nov 14 06:01:00.037 i-0c1eaa111e87f8a30 opal-client Trying server - ws://opal-server.local/ws
Nov 14 06:01:00.037 i-0c1eaa111e87f8a30 opal-client Trying to connect to Pub/Sub server - ws://opal-server.local/ws
Nov 14 06:01:00.037 i-0c1eaa111e87f8a30 opal-client Closing RPC client
Nov 14 06:01:00.037 i-0c1eaa111e87f8a30 opal-client Connection gracefully closed by server -- Trying to reconnect.
Nov 14 06:01:00.037 i-0c1eaa111e87f8a30 opal-client RPC Reader task was cancelled.
Nov 14 06:01:00.037 i-008606528e3b4be48 opal-client Trying server - ws://opal-server.local/ws
Nov 14 06:01:00.037 i-008606528e3b4be48 opal-client Trying to connect to Pub/Sub server - ws://opal-server.local/ws
Nov 14 06:01:00.037 i-008606528e3b4be48 opal-client Closing RPC client
Nov 14 06:01:00.037 i-008606528e3b4be48 opal-client Connection gracefully closed by server -- Trying to reconnect.
Nov 14 06:01:00.037 i-008606528e3b4be48 opal-client RPC Reader task was cancelled.
Nov 14 06:01:00.037 i-008606528e3b4be48 opal-client Disconnected from server
Nov 14 06:01:00.036 i-057b4106e35082453 opal-client Closing RPC client
Nov 14 06:01:00.036 i-057b4106e35082453 opal-client Connection gracefully closed by server -- Trying to reconnect.
Nov 14 06:01:00.036 i-057b4106e35082453 opal-client RPC Reader task was cancelled.
Nov 14 06:01:00.036 i-057b4106e35082453 opal-client Disconnected from server
Nov 14 06:01:00.036 i-057b4106e35082453 opal-client Closing RPC client
Nov 14 06:01:00.036 i-057b4106e35082453 opal-client Connection was terminated.
Nov 14 06:01:00.036 i-0c1eaa111e87f8a30 opal-client Disconnected from server
Nov 14 06:01:00.036 i-0c1eaa111e87f8a30 opal-client Closing RPC client
Nov 14 06:01:00.036 i-0c1eaa111e87f8a30 opal-client Connection was terminated.
Nov 14 06:01:00.036 i-008606528e3b4be48 opal-client Closing RPC client
Nov 14 06:01:00.036 i-008606528e3b4be48 opal-client Connection was terminated.
Nov 14 06:01:00.031 i-0dd6e5eac040e388e opal-client Trying server - ws://opal-server.local/ws
Nov 14 06:01:00.031 i-0dd6e5eac040e388e opal-client Trying to connect to Pub/Sub server - ws://opal-server.local/ws
Nov 14 06:01:00.031 i-0dd6e5eac040e388e opal-client Closing RPC client
Nov 14 06:01:00.030 i-0dd6e5eac040e388e opal-client Connection gracefully closed by server -- Trying to reconnect.
Nov 14 06:01:00.030 i-0dd6e5eac040e388e opal-client RPC Reader task was cancelled.
Nov 14 06:01:00.030 i-0dd6e5eac040e388e opal-client Disconnected from server
Nov 14 06:01:00.030 i-0dd6e5eac040e388e opal-client Closing RPC client
Nov 14 06:01:00.030 i-0dd6e5eac040e388e opal-client Connection was terminated.

OPAL-client memory increase at the same time the reconnection happens (Nov 14 06:01)

Expected behavior

• Memory should not accumulate across multiple reconnection events, as old data structures should be garbage collected after being replaced with new data
• OPAL server and client disconnect randomly/inconsistently without apparent reason, which should not happen

OPAL version

• Version: 0.9.0

[danyi1212]

Hey @goncalo-silva-prt , thank you for reaching out 🌟

When the OPAL client reconnects and performs the initial data load, it fetches data from the configured source and stores it in OPA's memory [1]. This is the normal operation of OPAL's data synchronization mechanism [2].

Here's what's happening:

Expected Behavior:

• When the OPAL client starts or reconnects, it performs a baseline data load to bring OPA from a no-state to a ready state [3]
• The data is fetched from your configured source (http://identity.local/api/users) and stored in OPA at the destination path (/users) [4]
• This data remains in memory as part of OPA's document hierarchy, which is necessary for authorization decisions [1]

Why Memory Increases on Reconnect:

• Each reconnection triggers a fresh "Initial load" that repopulates the data [3]
• If the data source has grown since the last load, the memory footprint will naturally increase
• The OPAL client stores this data in OPA's cache, which is designed to keep authorization data readily available in memory [2]

Regarding the Disconnections:
The periodic disconnections you're seeing could be related to:

• Network issues between client and server
• Kafka connection instability (given your SASL_SSL setup)
• Load balancer or ECS task health check configurations

The OPAL client will continue to try and fetch data until it succeeds, with exponential backoff [1]. This ensures eventual consistency even during temporary network issues.

Considering this is intended behavior, I'll be closing this issue. Feel free to continue discussion here 💎

[goncalo-silva-prt]

Hi @danyi1212,
Thanks for the explanation, but in our case we can confirm that the data source is not increasing between reconnections.

All the data stored in OPA comes only from the initial load.
No other data is being added to OPA after that point.

We verified this by restarting the ECS task: after the restart, the memory usage returns immediately to the normal baseline, so the memory increase is not caused by data growth on the source.

What we observe is:

• Every reconnection triggers a new “Initial load”, as expected
• After that log, memory usage increases by ~10%
• The dataset size remains exactly the same
• The memory never goes back down
• Restarting the task resets memory to the original level

This strongly indicates that something in the replace/update process for /users is leaving old objects in memory, which points to a memory leak rather than normal data growth.

[danyi1212]

Hey @goncalo-silva-prt, thank you for the details.

Based on your observation, I understand why it seem like a memory leak in that case.

1. What is the rough size in KB / MB your initial data load weighs?
2. After an "initial load" how many KB / MB does the RSS Memory increase by?

It is common that JSONs increase up to x20 compared to disk file (as documented here), so if the size increases by the memory of OPA in-memory storage, it could imply that the update method is related.

It is possible that some residue is kept somewhere inside the OPAL Client on the Python side, but there are minimal buffering or such that may have something weird going on.

If I may, why does your OPAL Client has so many reconnections? I would recommend checking the resources there, it may suggest something as well.

[goncalo-silva-prt]

Hi @danyi1212

1. The initial data load is ~13MB
2. The memory increases by ~80MB

When a reconnection happens, the memory increases by ~100MB.

Regarding the reconnections, I'm still unsure why they are occurring.
For additional context, I have multiple OPAL Client ECS tasks running, and 2 OPAL Server tasks running with UVICORN_NUM_WORKERS=4. Not all OPAL Client instances reconnect at the same time.
It seems that the clients which reconnect simultaneously are the ones connected to the same OPAL Server worker. This suggests that perhaps the worker is being restarted somehow, which causes all connected client instances to reconnect. However, I was expecting to see some indication of this in the OPAL Server logs, but nothing appears there.

[goncalo-silva-prt]

Hi @danyi1212 ,

Any update on this?
Could you please reopen this issue? The current behavior is not intended.

[danyi1212]

Hey @goncalo-silva-prt ,

We are investigating this issue, yes I'll reopen this issue until we settle this topic. I've opened an internal ticket as well for the investigation process. I'll make sure to update here about this topic.

Our current potential suspicious are CallbackReporter that may retain report data and OpaStaticDataCache that always copies the entire dataset on every write.