DuckDB
Introduction
Collects health, performance, and operational metrics for DuckDB embedded in a long-running microservice. Monitors query traffic, query latency, query failures, active queries, memory usage, Parquet load health, load failures, retries, last successful load time, and service readiness using Prometheus-compatible metrics exposed by the microservice.
Getting Started
Compatibility
DuckDB embedded inside a long-running microservice. Suitable for in-memory/read-only DuckDB deployments where the hosting service exposes Prometheus/OpenTelemetry metrics. Validated with Docker-based local deployment using service /metrics and /readyz endpoints.
Data Collection Method
vuSmartMaps collects DuckDB metrics from the hosting microservice's Prometheus-compatible /metrics endpoint using Telegraf inputs.prometheus. Optional readiness is collected from /readyz using Telegraf inputs.http_response. DuckDB is not monitored as a standalone database server.
Prerequisites
Inputs for Configuring Data Source
- Data Source Name: The display name for this DuckDB embedded service data source.
- DuckDB Service Base URL: Base URL of the microservice where DuckDB is embedded. Example: http://103.65.21.234:8081 or http://duckdb-service:8080. Do NOT include a trailing slash.
- Prometheus Metrics Endpoint URL: Full URL of the DuckDB microservice Prometheus metrics endpoint. Example: http://103.65.21.234:8081/metrics. If left empty in the template, the agent uses <server>/metrics.
- Poll Interval (seconds): How frequently DuckDB metrics and readiness are scraped. Recommended: 60–300 seconds.
- Authentication Method: Select the authentication method for the DuckDB embedded service metrics endpoint.
- Username: Username for Basic authentication. Shown only when Basic Auth is selected.
- Password / Token: Password or token for Basic authentication. Shown only when Basic Auth is selected.
Firewall Requirement
To collect data from this O11ySource, ensure the following ports are opened:
| Source IP | Destination IP | Destination Port | Protocol | Direction |
|---|---|---|---|---|
| vuSmartMaps IP | DuckDB embedded service / microservice | service metrics port, for example 8081 locally or the application's exposed port in Kubernetes | HTTP/HTTPS TCP | Inbound to the microservice metrics endpoint. |
Optional dependency monitoring: MinIO/S3 metrics can be collected separately if Parquet files are loaded from object storage.
*Before providing the firewall requirements, please update the port based on the customer environment.
Configuring the Target
Ensure the microservice exposing embedded DuckDB provides: /metrics - Prometheus/OpenTelemetry metrics endpoint /readyz - readiness endpoint, optional but recommended
Minimum expected metrics: duckdb_query_total duckdb_query_duration_seconds duckdb_active_queries duckdb_memory_bytes duckdb_load_duration_seconds duckdb_load_rows_total duckdb_load_files_total duckdb_last_load_timestamp duckdb_load_failures_total duckdb_load_retries_total
Verify access: curl http://:/readyz curl http://:/metrics | grep duckdb
Configuration Steps
- Add DuckDB as a data source in vuSmartMaps.
- Enter the DuckDB embedded service base URL, for example http://103.65.21.234:8081 or http://duckdb-service:8080.
- Enter the Prometheus metrics endpoint URL if different from /metrics.
- Select Authentication Method if the endpoint requires Basic Auth.
- Save — the agent scrapes /metrics using Telegraf Prometheus input and /readyz using HTTP response input.
Recommended: keep metrics labels such as target, instance, pod, namespace, service, and status for replica-level analysis.
Metrics Collected
| Name | Description | Data Type |
|---|---|---|
| @timestamp | Timestamp when the metric was collected | DateTime64 |
| kafka_topic | Name of the Kafka topic the record was read from | LowCardinality(String) |
| kafka_topic | Name of the Kafka topic the record was read from | String |
| host | Host that collected the metric | LowCardinality(String) |
| target | Target DuckDB service being monitored | LowCardinality(String) |
| target | Target DuckDB service being monitored | String |
| url | Endpoint URL that was scraped | String |
| timestamp | Detailed timestamp with milliseconds | DateTime64 |
| timestamp | Detailed timestamp with milliseconds | DateTime |
| content_length | Size of the HTTP response body in bytes | Nullable(UInt64) |
| http_response_code | HTTP response code returned by the endpoint | String |
| http_response_code | HTTP response code returned by the endpoint | Int64 |
| method | HTTP method used for the request | String |
| metric_type | Type of the collected metric record | String |
| status_code | Status code returned by the endpoint | String |
| response_time | Response time of the endpoint request | String |
| response_time | Response time of the endpoint request | Float64 |
| result | Result of the endpoint check | String |
| name | Name of the input / measurement that produced the record | String |
| result_code | Numeric code representing the result of the check | String |
| result_code | Numeric code representing the result of the check | Int64 |
| result_type | Type of the check result | String |
| server | Server address of the monitored endpoint | String |
| service | Name of the monitored service | String |
| python_gc_objects_collected_total | Objects collected during Python garbage collection | Nullable(Float64) |
| python_gc_objects_uncollectable_total | Uncollectable objects found during Python garbage collection | Nullable(Float64) |
| python_gc_collections_total | Number of times Python garbage collection has run | Nullable(Float64) |
| python_info | Python interpreter information of the microservice | Nullable(Float64) |
| process_virtual_memory_bytes | Virtual memory size of the process in bytes | Nullable(Float64) |
| process_resident_memory_bytes | Resident memory size of the process in bytes | Nullable(Float64) |
| process_start_time_seconds | Start time of the process since Unix epoch in seconds | Nullable(Float64) |
| process_cpu_seconds_total | Total user and system CPU time spent in seconds | Nullable(Float64) |
| process_open_fds | Number of open file descriptors | Nullable(Float64) |
| process_max_fds | Maximum number of open file descriptors | Nullable(Float64) |
| duckdb_query_total | Total number of queries executed on DuckDB | Nullable(Float64) |
| duckdb_query_duration_seconds | Duration of DuckDB queries in seconds | Nullable(Float64) |
| duckdb_query_duration_seconds_bucket | Query duration histogram bucket count (see le) | Nullable(Float64) |
| duckdb_query_duration_seconds_count | Number of observations in the query duration histogram | Nullable(Float64) |
| duckdb_query_duration_seconds_sum | Sum of all observed query durations in seconds | Nullable(Float64) |
| duckdb_query_duration_seconds_created | Creation timestamp of the query duration metric | Nullable(Float64) |
| duckdb_active_queries | Number of queries currently running on DuckDB | Nullable(Float64) |
| duckdb_slow_queries_total | Total number of slow queries | Nullable(Float64) |
| duckdb_slow_queries_created | Creation timestamp of the slow queries counter | Nullable(Float64) |
| duckdb_memory_bytes | Memory used by DuckDB in bytes | Nullable(Float64) |
| duckdb_load_duration_seconds | Duration of Parquet data loads in seconds | Nullable(Float64) |
| duckdb_load_duration_seconds_bucket | Load duration histogram bucket count (see le) | Nullable(Float64) |
| duckdb_load_duration_seconds_count | Number of observations in the load duration histogram | Nullable(Float64) |
| duckdb_load_duration_seconds_sum | Sum of all observed load durations in seconds | Nullable(Float64) |
| duckdb_load_duration_seconds_created | Creation timestamp of the load duration metric | Nullable(Float64) |
| duckdb_load_rows_total | Total number of rows loaded into DuckDB | Nullable(Float64) |
| duckdb_load_rows_created | Creation timestamp of the loaded rows counter | Nullable(Float64) |
| duckdb_load_files_total | Total number of files loaded into DuckDB | Nullable(Float64) |
| duckdb_load_files_created | Creation timestamp of the loaded files counter | Nullable(Float64) |
| duckdb_last_load_timestamp | Timestamp of the last successful data load | Nullable(Float64) |
| duckdb_load_failures_total | Total number of failed data loads | Nullable(Float64) |
| duckdb_load_failures_created | Creation timestamp of the load failures counter | Nullable(Float64) |
| dataset_freshness_seconds | Age of the loaded dataset in seconds | Nullable(Float64) |
| duckdb_load_retries_total | Total number of data load retries | Nullable(Float64) |
| duckdb_load_retries_created | Creation timestamp of the load retries counter | Nullable(Float64) |
| http_requests_total | Total number of HTTP requests served by the microservice | Nullable(Float64) |
| http_request_duration_seconds | Duration of HTTP requests in seconds | Nullable(Float64) |
| http_errors_total | Total number of HTTP errors | String |
| minio_fetch_duration_seconds | Duration of fetching data from MinIO in seconds | Nullable(Float64) |
| minio_fetch_duration_seconds_bucket | MinIO fetch duration histogram bucket count (see le) | Nullable(Float64) |
| minio_fetch_duration_seconds_count | Number of observations in the MinIO fetch duration histogram | Nullable(Float64) |
| minio_fetch_duration_seconds_sum | Sum of all observed MinIO fetch durations in seconds | Nullable(Float64) |
| minio_fetch_duration_seconds_created | Creation timestamp of the MinIO fetch duration metric | Nullable(Float64) |
| minio_read_failures_total | Total number of MinIO read failures | Nullable(Float64) |
| minio_read_failures_created | Creation timestamp of the MinIO read failures counter | Nullable(Float64) |
| le | Upper bound label of the histogram bucket | String |
| job | Prometheus job name | String |
| namespace | Kubernetes namespace of the service | String |
| pod | Kubernetes pod name of the service | String |
| status | Status label of the metric | String |
| instance | Instance (host:port) of the scraped endpoint | String |
