You shouldn't expect a database system to release memory back to the OS just because it's no longer needed for some specific purpose.

DuckDB has a memory_limit setting with a default of 80% of RAM. If you want to set a lower limit you can do something like

SET memory_limit = '1GB';

https://duckdb.org/2024/07/09/memory-management