Merge pull request #51 from benoitc/refactor/remove-auto-io-pool

Remove auto-started io pool to reduce memory usage

Author: benoitc

CHANGELOG.md

@@ -43,6 +43,11 @@
 
 ### Changed
 
+- **Removed auto-started io pool** - The io pool is no longer started automatically at
+  application startup to reduce memory usage. Users who need a dedicated I/O pool can
+  create one manually via `py_context_router:start_pool(io, 10, worker)`. The configuration
+  options `io_pool_size` and `io_pool_mode` have been removed.
+
 - **Removed py_event_router** - Removed legacy `py_event_router` module. The `py_event_worker`
   now handles all event loop functionality including FD events, timers, and task processing.
   This simplifies the architecture by consolidating event handling into a single worker process.
@@ -226,31 +231,22 @@
   - `py:dup_fd/1` - Duplicate fd for independent ownership
   - Prevents double-close issues when passing sockets to Python reactor
 
-- **Dual Pool Support** - Separate pools for CPU-bound and I/O-bound operations
-  - `default` pool - For quick CPU-bound operations, sized to number of schedulers
-  - `io` pool - For I/O-bound operations, larger pool (default: 10) for concurrency
-  - `py:call(io, Module, Func, Args)` - Execute on the io pool
-  - `py:call(io, Module, Func, Args, Kwargs)` - Execute with kwargs on io pool
-  - Registration-based routing (no call site changes needed):
-    - `py:register_pool(io, requests)` - Route all `requests.*` calls to io pool
-    - `py:register_pool(io, {aiohttp, get})` - Route specific function to io pool
-    - `py:unregister_pool(Module)` - Remove module registration
-    - `py:unregister_pool({Module, Func})` - Remove function registration
-    - Automatic routing: `py:call(requests, get, [Url])` goes to io pool when registered
+- **Custom Pool Support** - Create pools on demand for CPU-bound and I/O-bound operations
+  - `default` pool - Automatically started, sized to number of schedulers
   - `py_context_router:start_pool/2,3` - Start named pools programmatically
   - `py_context_router:stop_pool/1` - Stop a named pool
   - `py_context_router:pool_started/1` - Check if a pool is running
   - `py_context_router:get_context(Pool)` - Get context from a named pool
   - `py_context_router:num_contexts(Pool)` - Get pool size
   - `py_context_router:contexts(Pool)` - Get all contexts in a pool
   - `py_context_router:lookup_pool(Module, Func)` - Query pool routing
-  - Configuration via application env:
-    ```erlang
-    {erlang_python, [
-        {io_pool_size, 10},      % Size of io pool (default: 10)
-        {io_pool_mode, worker}   % Mode for io pool (default: auto)
-    ]}.
-    ```
+  - `py:call(PoolName, Module, Func, Args)` - Execute on a specific pool
+  - Registration-based routing (no call site changes needed):
+    - `py:register_pool(io, requests)` - Route all `requests.*` calls to io pool
+    - `py:register_pool(io, {aiohttp, get})` - Route specific function to io pool
+    - `py:unregister_pool(Module)` - Remove module registration
+    - `py:unregister_pool({Module, Func})` - Remove function registration
+    - Automatic routing: `py:call(requests, get, [Url])` goes to io pool when registered
   - Backward compatible: existing code using `py:call/3,4,5` works unchanged
   - New test suite: `test/py_pool_SUITE.erl`
 

docs/distributed.md

@@ -45,7 +45,7 @@ erlang_python integrates with Erlang's distribution to run Python code on remote
 | Async tasks | `rpc:call(Node, py_event_loop, create_task, ...)` |
 | Venv management | `py:ensure_venv/2,3` |
 | Data streaming | `py_channel` API |
-| Pool routing | Dual pool (default/io) |
+| Pool routing | Custom pools on demand |
 
 ## Basic Remote Execution
 
@@ -496,9 +496,6 @@ Configure nodes for distributed Python:
         %% More contexts for dedicated Python workers
         {num_contexts, 16},
 
-        %% Larger IO pool for network operations
-        {io_pool_size, 20},
-
         %% Higher concurrency limit
         {max_concurrent, 100}
     ]},

docs/getting-started.md

@@ -466,11 +466,14 @@ end.
 
 See [Security](security.md) for details on blocked operations and recommended alternatives.
 
-## Dual Pool Support
+## Custom Pool Support
 
-erlang_python provides two pools to separate CPU-bound and I/O-bound operations:
+erlang_python lets you create pools on demand to separate CPU-bound and I/O-bound operations:
 
 ```erlang
+%% Create io pool for I/O-bound operations
+{ok, _} = py_context_router:start_pool(io, 10, worker).
+
 %% Register entire modules to io pool
 py:register_pool(io, requests).
 py:register_pool(io, psycopg2).
@@ -484,7 +487,7 @@ py:register_pool(io, {db, query}).        %% Only db.query goes to io pool
 {ok, Rows} = py:call(db, query, [Sql]).      %% -> io pool (callable registered)
 ```
 
-This prevents slow HTTP requests from blocking quick math operations. See [Dual Pool Support](pools.md) for configuration and advanced usage.
+This prevents slow HTTP requests from blocking quick math operations. See [Pool Support](pools.md) for configuration and advanced usage.
 
 ## Zero-Copy Buffers
 
@@ -517,7 +520,7 @@ See [Buffer API](buffer.md) for zero-copy memoryview access and fast substring s
 
 ## Next Steps
 
-- See [Dual Pool Support](pools.md) for separating CPU and I/O operations
+- See [Pool Support](pools.md) for separating CPU and I/O operations
 - See [Type Conversion](type-conversion.md) for detailed type mapping
 - See [Context Affinity](context-affinity.md) for preserving Python state
 - See [Streaming](streaming.md) for working with generators

docs/migration.md

@@ -533,15 +533,16 @@ ok = py:deactivate_venv().
 {ok, #{<<"active">> := true, <<"venv_path">> := Path}} = py:venv_info().
 ```
 
-### Dual Pool Support
+### Custom Pool Support
 
-Separate pools for CPU-bound and I/O-bound operations:
+Create pools on demand for CPU-bound and I/O-bound operations:
 
 ```erlang
 %% Default pool - CPU-bound operations (sized to schedulers)
 {ok, Result} = py:call(math, sqrt, [16]).
 
-%% IO pool - I/O-bound operations (larger pool, default 10)
+%% Create io pool for I/O-bound operations
+{ok, _} = py_context_router:start_pool(io, 10, worker).
 {ok, Response} = py:call(io, requests, get, [Url]).
 
 %% Registration-based routing (no call site changes)
@@ -552,14 +553,6 @@ py:register_pool(io, {aiohttp, get}),        % Route specific function
 {ok, Response} = py:call(requests, get, [Url]).  % Goes to io pool
 ```
 
-Configuration in `sys.config`:
-```erlang
-{erlang_python, [
-    {io_pool_size, 10},     % Size of io pool (default: 10)
-    {io_pool_mode, worker}  % Mode for io pool (default: auto)
-]}.
-```
-
 ## Performance Improvements
 
 The v2.0 release includes significant performance improvements:

docs/pools.md

@@ -1,17 +1,17 @@
-# Dual Pool Support
+# Pool Support
 
-This guide covers erlang_python's dual pool architecture for separating CPU-bound and I/O-bound Python operations.
+This guide covers erlang_python's pool architecture for separating CPU-bound and I/O-bound Python operations.
 
 ## Overview
 
-erlang_python provides two separate pools of Python contexts:
+erlang_python provides a `default` pool that starts automatically, and allows you to create additional pools on demand:
 
 | Pool | Purpose | Default Size | Use Case |
 |------|---------|--------------|----------|
 | `default` | Quick CPU-bound operations | Number of schedulers | Math, string processing, data transformation |
-| `io` | Slow I/O-bound operations | 10 | HTTP requests, database queries, file I/O |
+| custom pools | User-defined pools | User-defined | HTTP requests, database queries, GPU work |
 
-This separation prevents slow I/O operations from blocking quick CPU operations.
+Create pools on demand to separate slow I/O operations from blocking quick CPU operations.
 
 ## Architecture
 
@@ -28,19 +28,31 @@ This separation prevents slow I/O operations from blocking quick CPU operations.
 │              ┌──────────────┴──────────────┐                     │
 │              ▼                              ▼                     │
 │     ┌────────────────┐             ┌────────────────┐            │
-│     │  default pool  │             │    io pool     │            │
-│     │  (N contexts)  │             │  (10 contexts) │            │
+│     │  default pool  │             │  custom pools  │            │
+│     │  (N contexts)  │             │  (on demand)   │            │
 │     └────────────────┘             └────────────────┘            │
 │              │                              │                     │
 │     ┌────────┴────────┐            ┌────────┴────────┐           │
 │     ▼        ▼        ▼            ▼        ▼        ▼           │
-│   Ctx1    Ctx2    CtxN          Ctx1    Ctx2    Ctx10           │
+│   Ctx1    Ctx2    CtxN          Ctx1    Ctx2    CtxN            │
 │   (math)  (json)  (...)         (http)  (db)   (...)            │
 └──────────────────────────────────────────────────────────────────┘
 ```
 
 ## Basic Usage
 
+### Creating Custom Pools
+
+Create pools on demand for specific workloads:
+
+```erlang
+%% Create an io pool for I/O-bound operations
+{ok, _Contexts} = py_context_router:start_pool(io, 10, worker).
+
+%% Create a gpu pool for ML workloads
+{ok, _} = py_context_router:start_pool(gpu, 2, worker).
+```
+
 ### Explicit Pool Selection
 
 Specify the pool directly in the call:
@@ -49,7 +61,7 @@ Specify the pool directly in the call:
 %% Use default pool (quick operations)
 {ok, 4.0} = py:call(default, math, sqrt, [16]).
 
-%% Use io pool (slow operations)
+%% Use io pool (after creating it)
 {ok, Response} = py:call(io, requests, get, [Url]).
 
 %% With keyword arguments
@@ -153,31 +165,30 @@ default = py_context_router:lookup_pool(json, dumps).  %% Function override
 
 ## Configuration
 
-Configure pool sizes via application environment:
+Configure default pool size via application environment:
 
 ```erlang
 %% sys.config
 [
     {erlang_python, [
         %% Default pool size (default: erlang:system_info(schedulers))
-        {default_pool_size, 8},
-
-        %% IO pool size (default: 10)
-        {io_pool_size, 20},
-
-        %% IO pool mode: auto | subinterp | worker (default: auto)
-        {io_pool_mode, worker}
+        {default_pool_size, 8}
     ]}
 ].
 ```
 
 ### Runtime Configuration
 
 ```erlang
-%% Start additional custom pool
+%% Start io pool for I/O-bound operations
+{ok, _} = py_context_router:start_pool(io, 10, worker).
+
+%% Start GPU pool for ML operations
 {ok, _} = py_context_router:start_pool(gpu, 2, worker).
 
-%% Register GPU operations
+%% Register operations to route to specific pools
+ok = py:register_pool(io, requests).
+ok = py:register_pool(io, psycopg2).
 ok = py:register_pool(gpu, torch).
 ok = py:register_pool(gpu, tensorflow).
 ```
@@ -189,6 +200,9 @@ ok = py:register_pool(gpu, tensorflow).
 ```erlang
 %% At application startup
 init_pools() ->
+    %% Create io pool for I/O-bound operations
+    {ok, _} = py_context_router:start_pool(io, 10, worker),
+
     %% Register I/O-heavy modules
     py:register_pool(io, requests),
     py:register_pool(io, httpx),
@@ -212,7 +226,8 @@ handle_request(UserId) ->
 ### ML Pipeline with I/O
 
 ```erlang
-%% Register I/O operations
+%% Create and configure io pool
+{ok, _} = py_context_router:start_pool(io, 10, worker),
 py:register_pool(io, boto3),        %% S3 access
 py:register_pool(io, requests),      %% API calls
 
@@ -254,11 +269,15 @@ process_batch(Items) ->
 
 ```erlang
 %% Check pool status
-true = py_context_router:pool_started(default),
+true = py_context_router:pool_started(default).
+false = py_context_router:pool_started(io).  %% Not started yet
+
+%% Start io pool
+{ok, _} = py_context_router:start_pool(io, 10, worker).
 true = py_context_router:pool_started(io).
 
 %% Check pool sizes
-DefaultSize = py_context_router:num_contexts(default),
+DefaultSize = py_context_router:num_contexts(default).
 IoSize = py_context_router:num_contexts(io).
 
 %% List all registrations

src/py_context_init.erl

@@ -15,21 +15,18 @@
 %%% @doc Initializes the context router during application startup.
 %%%
 %%% This module provides a supervisor-compatible start function that
-%%% initializes the context pools and returns `ignore' (since no
+%%% initializes the context pool and returns `ignore' (since no
 %%% process needs to stay running after initialization).
 %%%
 %%% == Pools ==
 %%%
-%%% Two pools are started by default:
-%%% - `default' - For quick CPU-bound operations, sized to number of schedulers
-%%% - `io' - For I/O-bound operations, larger pool (default: 10) for concurrency
+%%% The `default' pool is started automatically, sized to number of schedulers.
+%%% Additional pools can be created on demand via `py_context_router:start_pool/3'.
 %%%
-%%% Pool sizes can be configured via application env:
+%%% Pool size can be configured via application env:
 %%% ```
 %%% {erlang_python, [
-%%%     {default_pool_size, 4},        % Number of contexts (default: schedulers)
-%%%     {io_pool_size, 10},            % I/O pool size (default: 10)
-%%%     {io_pool_mode, worker}         % Mode for io pool (default: worker)
+%%%     {default_pool_size, 4}         % Number of contexts (default: schedulers)
 %%% ]}.
 %%% '''
 %%% @private
@@ -53,17 +50,9 @@ start_link(Opts) ->
 
     case py_context_router:start_pool(default, DefaultSize, DefaultMode) of
         {ok, _DefaultContexts} ->
-            %% Start I/O pool if configured
-            IoSize = application:get_env(erlang_python, io_pool_size, 10),
-            IoMode = application:get_env(erlang_python, io_pool_mode, worker),
-            case py_context_router:start_pool(io, IoSize, IoMode) of
-                {ok, _IoContexts} ->
-                    %% The contexts are supervised by py_context_sup
-                    %% We don't need a process here, just return ignore
-                    ignore;
-                {error, IoReason} ->
-                    {error, {io_pool_start_failed, IoReason}}
-            end;
+            %% The contexts are supervised by py_context_sup
+            %% We don't need a process here, just return ignore
+            ignore;
         {error, Reason} ->
             {error, {default_pool_start_failed, Reason}}
     end.

test/py_pool_SUITE.erl

@@ -1,6 +1,6 @@
-%%% @doc Common Test suite for dual pool support.
+%%% @doc Common Test suite for pool support.
 %%%
-%%% Tests the default and io pool separation to ensure:
+%%% Tests the default pool and custom pools to ensure:
 %%% - Pools are independent
 %%% - Pool-based calls work correctly
 %%% - Backward compatibility is maintained
@@ -58,6 +58,8 @@ all() ->
 
 init_per_suite(Config) ->
     application:ensure_all_started(erlang_python),
+    %% Create io pool manually (no longer auto-started)
+    {ok, _} = py_context_router:start_pool(io, 10, worker),
     Config.
 
 end_per_suite(_Config) ->
@@ -73,15 +75,15 @@ end_per_testcase(_TestCase, _Config) ->
 %% Test cases
 %% ============================================================================
 
-%% @doc Verify both pools are started on application start.
+%% @doc Verify pools are started (default auto, io from init_per_suite).
 test_pools_started(_Config) ->
-    %% Default pool should be started
+    %% Default pool should be started automatically
     true = py_context_router:pool_started(default),
     DefaultSize = py_context_router:num_contexts(default),
     true = DefaultSize > 0,
     ct:pal("Default pool size: ~p", [DefaultSize]),
 
-    %% IO pool should be started
+    %% IO pool was started in init_per_suite
     true = py_context_router:pool_started(io),
     IoSize = py_context_router:num_contexts(io),
     true = IoSize > 0,
@@ -191,10 +193,10 @@ test_pool_sizes(_Config) ->
     ExpectedDefault = erlang:system_info(schedulers),
     ct:pal("Default pool: expected ~p, actual ~p", [ExpectedDefault, DefaultSize]),
 
-    %% IO pool should be 10 by default (from py_context_init)
+    %% IO pool was created with size 10 in init_per_suite
     IoSize = py_context_router:num_contexts(io),
     ct:pal("IO pool size: ~p", [IoSize]),
-    true = IoSize > 0,
+    10 = IoSize,
 
     %% Verify we can list all contexts from each pool
     DefaultContexts = py_context_router:contexts(default),