Skip to main content
Version: NG-2.16

DuckDB

Introduction​

Collects health, performance, and operational metrics for DuckDB embedded in a long-running microservice. Monitors query traffic, query latency, query failures, active queries, memory usage, Parquet load health, load failures, retries, last successful load time, and service readiness using Prometheus-compatible metrics exposed by the microservice.

Getting Started​

Compatibility​

DuckDB embedded inside a long-running microservice. Suitable for in-memory/read-only DuckDB deployments where the hosting service exposes Prometheus/OpenTelemetry metrics. Validated with Docker-based local deployment using service /metrics and /readyz endpoints.

Data Collection Method​

vuSmartMaps collects DuckDB metrics from the hosting microservice's Prometheus-compatible /metrics endpoint using Telegraf inputs.prometheus. Optional readiness is collected from /readyz using Telegraf inputs.http_response. DuckDB is not monitored as a standalone database server.

Prerequisites​

Inputs for Configuring Data Source​

  • Data Source Name: The display name for this DuckDB embedded service data source.
  • DuckDB Service Base URL: Base URL of the microservice where DuckDB is embedded. Example: http://103.65.21.234:8081 or http://duckdb-service:8080. Do NOT include a trailing slash.
  • Prometheus Metrics Endpoint URL: Full URL of the DuckDB microservice Prometheus metrics endpoint. Example: http://103.65.21.234:8081/metrics. If left empty in the template, the agent uses <server>/metrics.
  • Poll Interval (seconds): How frequently DuckDB metrics and readiness are scraped. Recommended: 60–300 seconds.
  • Authentication Method: Select the authentication method for the DuckDB embedded service metrics endpoint.
  • Username: Username for Basic authentication. Shown only when Basic Auth is selected.
  • Password / Token: Password or token for Basic authentication. Shown only when Basic Auth is selected.

Firewall Requirement​

To collect data from this O11ySource, ensure the following ports are opened:

Source IPDestination IPDestination PortProtocolDirection
vuSmartMaps IPDuckDB embedded service / microserviceservice metrics port, for example 8081 locally or the application's exposed port in KubernetesHTTP/HTTPS TCPInbound to the microservice metrics endpoint.

Optional dependency monitoring: MinIO/S3 metrics can be collected separately if Parquet files are loaded from object storage.

*Before providing the firewall requirements, please update the port based on the customer environment.

Configuring the Target​

Ensure the microservice exposing embedded DuckDB provides: /metrics - Prometheus/OpenTelemetry metrics endpoint /readyz - readiness endpoint, optional but recommended

Minimum expected metrics: duckdb_query_total duckdb_query_duration_seconds duckdb_active_queries duckdb_memory_bytes duckdb_load_duration_seconds duckdb_load_rows_total duckdb_load_files_total duckdb_last_load_timestamp duckdb_load_failures_total duckdb_load_retries_total

Verify access: curl http://:/readyz curl http://:/metrics | grep duckdb

Configuration Steps​

  1. Add DuckDB as a data source in vuSmartMaps.
  2. Enter the DuckDB embedded service base URL, for example http://103.65.21.234:8081 or http://duckdb-service:8080.
  3. Enter the Prometheus metrics endpoint URL if different from /metrics.
  4. Select Authentication Method if the endpoint requires Basic Auth.
  5. Save — the agent scrapes /metrics using Telegraf Prometheus input and /readyz using HTTP response input.

Recommended: keep metrics labels such as target, instance, pod, namespace, service, and status for replica-level analysis.

Metrics Collected​

NameDescriptionData Type
@timestampTimestamp when the metric was collectedDateTime64
kafka_topicName of the Kafka topic the record was read fromLowCardinality(String)
kafka_topicName of the Kafka topic the record was read fromString
hostHost that collected the metricLowCardinality(String)
targetTarget DuckDB service being monitoredLowCardinality(String)
targetTarget DuckDB service being monitoredString
urlEndpoint URL that was scrapedString
timestampDetailed timestamp with millisecondsDateTime64
timestampDetailed timestamp with millisecondsDateTime
content_lengthSize of the HTTP response body in bytesNullable(UInt64)
http_response_codeHTTP response code returned by the endpointString
http_response_codeHTTP response code returned by the endpointInt64
methodHTTP method used for the requestString
metric_typeType of the collected metric recordString
status_codeStatus code returned by the endpointString
response_timeResponse time of the endpoint requestString
response_timeResponse time of the endpoint requestFloat64
resultResult of the endpoint checkString
nameName of the input / measurement that produced the recordString
result_codeNumeric code representing the result of the checkString
result_codeNumeric code representing the result of the checkInt64
result_typeType of the check resultString
serverServer address of the monitored endpointString
serviceName of the monitored serviceString
python_gc_objects_collected_totalObjects collected during Python garbage collectionNullable(Float64)
python_gc_objects_uncollectable_totalUncollectable objects found during Python garbage collectionNullable(Float64)
python_gc_collections_totalNumber of times Python garbage collection has runNullable(Float64)
python_infoPython interpreter information of the microserviceNullable(Float64)
process_virtual_memory_bytesVirtual memory size of the process in bytesNullable(Float64)
process_resident_memory_bytesResident memory size of the process in bytesNullable(Float64)
process_start_time_secondsStart time of the process since Unix epoch in secondsNullable(Float64)
process_cpu_seconds_totalTotal user and system CPU time spent in secondsNullable(Float64)
process_open_fdsNumber of open file descriptorsNullable(Float64)
process_max_fdsMaximum number of open file descriptorsNullable(Float64)
duckdb_query_totalTotal number of queries executed on DuckDBNullable(Float64)
duckdb_query_duration_secondsDuration of DuckDB queries in secondsNullable(Float64)
duckdb_query_duration_seconds_bucketQuery duration histogram bucket count (see le)Nullable(Float64)
duckdb_query_duration_seconds_countNumber of observations in the query duration histogramNullable(Float64)
duckdb_query_duration_seconds_sumSum of all observed query durations in secondsNullable(Float64)
duckdb_query_duration_seconds_createdCreation timestamp of the query duration metricNullable(Float64)
duckdb_active_queriesNumber of queries currently running on DuckDBNullable(Float64)
duckdb_slow_queries_totalTotal number of slow queriesNullable(Float64)
duckdb_slow_queries_createdCreation timestamp of the slow queries counterNullable(Float64)
duckdb_memory_bytesMemory used by DuckDB in bytesNullable(Float64)
duckdb_load_duration_secondsDuration of Parquet data loads in secondsNullable(Float64)
duckdb_load_duration_seconds_bucketLoad duration histogram bucket count (see le)Nullable(Float64)
duckdb_load_duration_seconds_countNumber of observations in the load duration histogramNullable(Float64)
duckdb_load_duration_seconds_sumSum of all observed load durations in secondsNullable(Float64)
duckdb_load_duration_seconds_createdCreation timestamp of the load duration metricNullable(Float64)
duckdb_load_rows_totalTotal number of rows loaded into DuckDBNullable(Float64)
duckdb_load_rows_createdCreation timestamp of the loaded rows counterNullable(Float64)
duckdb_load_files_totalTotal number of files loaded into DuckDBNullable(Float64)
duckdb_load_files_createdCreation timestamp of the loaded files counterNullable(Float64)
duckdb_last_load_timestampTimestamp of the last successful data loadNullable(Float64)
duckdb_load_failures_totalTotal number of failed data loadsNullable(Float64)
duckdb_load_failures_createdCreation timestamp of the load failures counterNullable(Float64)
dataset_freshness_secondsAge of the loaded dataset in secondsNullable(Float64)
duckdb_load_retries_totalTotal number of data load retriesNullable(Float64)
duckdb_load_retries_createdCreation timestamp of the load retries counterNullable(Float64)
http_requests_totalTotal number of HTTP requests served by the microserviceNullable(Float64)
http_request_duration_secondsDuration of HTTP requests in secondsNullable(Float64)
http_errors_totalTotal number of HTTP errorsString
minio_fetch_duration_secondsDuration of fetching data from MinIO in secondsNullable(Float64)
minio_fetch_duration_seconds_bucketMinIO fetch duration histogram bucket count (see le)Nullable(Float64)
minio_fetch_duration_seconds_countNumber of observations in the MinIO fetch duration histogramNullable(Float64)
minio_fetch_duration_seconds_sumSum of all observed MinIO fetch durations in secondsNullable(Float64)
minio_fetch_duration_seconds_createdCreation timestamp of the MinIO fetch duration metricNullable(Float64)
minio_read_failures_totalTotal number of MinIO read failuresNullable(Float64)
minio_read_failures_createdCreation timestamp of the MinIO read failures counterNullable(Float64)
leUpper bound label of the histogram bucketString
jobPrometheus job nameString
namespaceKubernetes namespace of the serviceString
podKubernetes pod name of the serviceString
statusStatus label of the metricString
instanceInstance (host:port) of the scraped endpointString