2 posts tagged with "Upgrade"

Overview​

vuSmartMaps™ NG 3.2.1 is a patch release of NG 3.2.1 that strengthens the ‘NG 2.16 to NG 3.2.1 Traces migration workflow’, introducing backward compatibility, rollback support, CPU capability validation, and improved trace configuration handling. This release enables seamless migration of existing traces from legacy dashboards to APM Smart Frames, where legacy dashboards continue to work as-is while Smart Frames provide access to latest APM data.

For air-gapped environments without external registry access, NG 3.2.1 introduces ‘enhanced local registry support’ that eliminates the dependency on GHCR or external registries. Helm charts can now be uploaded and managed through a configured local registry, ensuring seamless upgrades in disconnected environments.

Additionally, the release includes enhancements for O11ySource configuration, OmniAgent performance, Kubernetes security hardening, and platform deployment improvements. The release also resolves multiple issues affecting ADI discovery probe handling, O11ySource configuration and deployment, APM and distributed tracing, data model permissions, OmniAgent installation on Windows, log forwarding on AIX, and upgrade and migration workflows. These comprehensive improvements enhance platform stability, security, configuration management, migration continuity, and overall operational reliability.

Traces Migration​

CPU Capability Validation in Upgrade Pre-Checks​

Enhanced the pre-checks for NG 2.16 to NG 3.2.1 upgrade and migration by adding CPU capability validation before installation or upgrade begins. The pre-checks now verify the CPU instruction-set capabilities required for the x86-64-v3 compatibility level, including AVX, AVX2, BMI1, BMI2, F16C, FMA, LZCNT, and MOVBE. If the target server or VM does not support the required capabilities, the pre-checks clearly report the CPU compatibility issue before the upgrade or migration starts.

• CCBD-5709: Add CPU Capability Validation to Pre-checks Script

Host and Application Information in Trace Resource Attributes​

Enhanced trace instrumentation during the NG 2.16 to NG 3.2.1 migration by adding the source host IP and application name to the OpenTelemetry resource attributes. The instrumentation now includes the source host IP and resolves configured application IDs to application names, allowing the corresponding host and application information to be associated with trace data. The configuration update also ensures that changes to the applications associated with an instrumentation source automatically trigger OmniAgent configuration regeneration and deployment.

This ensures that updated trace instrumentation configurations are correctly reflected in the deployed OmniAgent configuration during the Traces migration.

• CCBD-5747: Add host_ip and application to OTEL_RESOURCE_ATTRIBUTES and trigger OmniAgent config regeneration when awb_apps changes

Preserve OTEL Collector Configuration During Platform Upgrade​

Updated the platform upgrade process to skip updating the OTEL Collector deployment template during the platform upgrade step. This ensures that Traces continues to use the existing O11ySource configuration after the platform upgrade. The OTEL Collector deployment template is updated separately as part of the Traces upgrade activity, preventing the platform upgrade from unintentionally affecting the existing Traces configuration.

• CCBD-5764: Skip OTEL Collector Deployment Template Update During Platform Upgrade

Normalize Semantic Convention Attributes for Traces Migration​

Fixed semantic convention attribute normalization for Traces migration by extending the mapping to support attributes emitted by C++ and NodeJS agents. The updated normalization maps pre-stable attributes to their corresponding standard attributes, ensuring required information such as network, client, server, and port attributes is available for Traces processing and related visualizations.

• CCBD-5763: Normalize semconv attributes for phased Traces migration with new entries

Traces Migration Backward Compatibility for NG 2.16​

Added a Traces migration script to support backward compatibility with the NG 2.16 Traces implementation during migration to NG 3.2.1. The migration process applies the required compatibility changes so that existing environments can continue using the old SDK while also supporting the new SDK after migration. This helps maintain existing Traces dashboards and data flow during the transition.

• Add traces_migration.py for NG 2.16 backward compatibility during Traces migration

Rollback Support for Traces Migration​

Enhanced the Traces migration process with rollback support, allowing migration changes to be reverted if an issue occurs during or after migration. The rollback mechanism restores the environment to its state before the migration and helps prevent partial migration or schema inconsistencies. It also supports rollback dry runs and automatic rollback control.

Minor Enhancements​

OmniAgent Enhancements​

Disable Log Collection to Improve OmniAgent Performance​

OmniAgent performance has been improved by disabling log collection to reduce the overhead associated with log collection.

• CCBD-5693: Disable log collection to improve performance of OmniAgent

Security Enhancements​

Server-Side Validation for Event Export APIs​

Enhanced security for event export APIs by adding server-side validation for special characters that could be used in CSV or formula injection attacks. Previously, validation was performed only on the frontend and could be bypassed by sending requests directly to the backend. The backend now enforces the same validation rules and rejects requests containing disallowed special characters, preventing validation bypass through direct API requests and improving data integrity.

• CCBD-5397: Prevent CSV/Formula Injection in Event Export APIs

Kubernetes Security Hardening for RUM Collector​

Enhanced the RUM Collector deployment with Kubernetes security controls to provide a more restricted and secure runtime environment. The collector now runs as a non-root user with a read-only root filesystem, disabled privilege escalation, dropped Linux capabilities, and the RuntimeDefault seccomp profile. AppArmor runtime-default protection and disabled automatic service-account token mounting provide additional security by limiting unnecessary system and Kubernetes API access. These changes reduce the attack surface and strengthen the overall security of the RUM Collector deployment.

• CCBD-5099: [K8s Hardening] Add full securityContext to rum-collector deployment

Kubernetes Security Hardening for Local Registry Components​

Enhanced the security of the vunet-registry, image-update-job, and cert-update-local-registry Helm charts by strengthening their Kubernetes security configurations. The workloads now use non-privileged containers with privilege escalation disabled, dropped Linux capabilities, and read-only root filesystems. Unnecessary host access to /etc/kubernetes has also been removed, and the registry deployment no longer automatically mounts a Kubernetes service-account token.

Resource configuration has also been applied to the image update job to provide better control over CPU and memory usage. These changes reduce unnecessary privileges and host access while strengthening the security of local registry components.

• CCBD-4792: Harden vunet-registry chart family

Platform / Deployment Enhancements​

Local Registry Support for Air-Gapped Upgrades​

Enhanced local registry support for air-gapped environments by defining the Helm chart upload path for upgrade scenarios that use a local registry. The upgrade workflow now supports uploading the required Helm charts to the configured local registry, eliminating the dependency on an externally configured registry for these environments. The registry pull-secret handling has also been updated to support the required https:// prefix, with cleaner registry push output during the process.

• CCBD-5514: Registry pull-secret requires https:// prefix and cleaner push output

Deployment Resource Limits Validation​

Enhanced deployment validation to verify that resource limits configured for critical platform services match the sizing defined in the deployment specification. The validation now checks the CPU and memory limits for critical services, including ClickHouse, Kafka, Nairobi, Denver, OTEL Collector, ZooKeeper, ClickHouse Keeper, PostgreSQL, vuInterface, Renderer, and Reporter. It also verifies that configured limits do not fall below the defined minimum requirements. If a deployment does not have a sizing specification, the validation uses the defined minimum limits. The audit also flags sizing specifications that define resource limits below the required minimum values.

• CCBD-2439: CH and CH Keeper Pod Limits Validation and Minimum Limits Check

Automated Kubernetes Cluster Upgrade with vuKube

Enhanced the vuKube upgrade workflow with vukube-upgrade, an automated utility for upgrading Kubernetes clusters during migration to vuKube. The utility supports the vuKube upgrade workflow and helps simplify and automate cluster upgrade operations.

• CCBD-5371: Add vukube-upgrade – automated Kubernetes cluster upgrade tool

Bug Fixes​

ADI Bug fixes​

Discovery Probe Handling for Hosts Without Instrumentation​

Enhanced ADI discovery probe handling for hosts where OmniAgent was installed before instrumentation was enabled. The discovery manager now checks the host's instrumentation status before deploying a discovery probe. Probes are deployed only when instrumentation is successfully applied. Hosts where instrumentation is not yet applied or has failed are deferred and retried automatically after the required instrumentation is enabled.

Hosts that do not support instrumentation, such as unsupported platforms, are skipped without entering repeated retry cycles. The OmniAgent installation process also ensures that newly created directories have the correct ownership, allowing the service to create the required instrumentation directory.

• CCBD-5771: Stop the discovery probe from failing on hosts installed before ADI was enabled

Exclude JVM Agent Flags from Service Fingerprint​

Updated ADI Java service discovery to exclude JVM agent attachment flags from the service fingerprint. This ensures that JVM agent attachment flags are not considered when generating the service fingerprint.

• VUQA-7125: Exclude JVM agent attachment flags from the service fingerprint

O11ySource Bug Fixes​

Windows Log Path Validation Fix for MySQL and MariaDB​

Fixed an issue where valid Windows file paths for the MySQL and MariaDB Slow Log Path and Error Log Path fields were rejected by O11ySource validation. The path validation pattern has been corrected to properly recognize Windows-style paths while continuing to support Linux paths. This allows valid Windows log file paths to be accepted when configuring MySQL and MariaDB O11ySources.

• CCBD-5784: Fix Windows log path validation regex for MySQL/MariaDB

GHCR Authentication for Telegraf O11ySources​

Fixed an issue where the ghcr-login-secret was missing from certain Telegraf-based O11ySource deployment templates. The affected templates now include the required image-pull secret, allowing Telegraf O11ySource pods to authenticate with GitHub Container Registry (GHCR) and successfully pull their container images.

• VUQA-7149: Add ghcr-login-secret to missing Telegraf O11ySource templates

Telegraf O11ySource Pods Stuck in Pending State​

Fixed an issue where Telegraf-based O11ySource pods could remain in the Pending state on Kubernetes clusters where nodes are configured with taints. The deployment configuration now derives the appropriate node placement tags from the deployment specification and applies them to Telegraf O11ySources. This allows Telegraf-based O11ySource pods to be scheduled on the appropriate nodes in tainted-node environments.

• CCBD-5801: Telegraf-based O11ySource pods stuck in Pending on tainted-node clusters

Tomcat O11ySource Configuration Preservation During Migration​

Fixed an issue where Tomcat O11ySource configuration values such as port, ssl, application_metrics, and application_logs were not preserved correctly during OmniAgent migration. The Tomcat templates and migration handling have been updated to correctly retain these configuration values from the existing Tomcat setup. This prevents the migrated Tomcat O11ySource from losing required configuration and ensures that JMX metrics and application log collection continue to work after migration.

• VUQA-7159: Fix Tomcat Jinja templates dropping ssl, application_metrics, and application_logs

Clear Hidden O11ySource Fields When Dependencies Are Disabled​

Fixed an issue where hidden dependent fields in the Add/Edit O11ySource form could retain their previous values after the controlling option was disabled. For example, when TLS is disabled for a MongoDB O11ySource, dependent fields such as TLS Mode, Skip Certificate Verification, and TLS Credential are now cleared instead of retaining their previously saved values. This ensures that the saved O11ySource configuration reflects the options currently enabled.

• VUQA-7164: O11ySources send rule-hidden fields as null instead of their stale values

Cassandra Healthbeat Module Configuration Path Fix​

Fixed an issue in the Cassandra O11ySource where the Healthbeat modules.d path used an incorrect Windows-style path separator. The configuration now uses the correct Linux-style path separator, allowing Healthbeat to locate and load the Cassandra module configuration files successfully. This restores the expected Cassandra data ingestion and dashboard data.

• CCBD-5838: Fix Cassandra O11ySource Healthbeat modules.d path

Data Model / Permissions Fixes​

Data Model Object-Level Permission Fix​

Fixed an issue where users with access to the Data Modelling Workspace could access data models for which they had not been explicitly granted permissions. Object-level permission checks are now enforced for individual data models. Users can access only the data models for which they have the required permissions, both through the Data Modelling Workspace and VED.

• VUQA-7123: Add single-object permission check to ObjectPermissionsFilterBackend

Data Model Threshold Error in Smart Frame Studio​

Fixed an issue where creating a Smart Frame with a data model containing a threshold resulted in a 500 Internal Server Error. Smart Frames can now be created successfully using data models with or without thresholds.

• VUQA-7148: Data Model Thresholds returning 500 Internal Server Error in SF Panels

APM / Tracing Bug Fixes​

OTEL Collector status_code Fix for Traces Backward Compatibility​

Fixed an issue in the OTEL Collector where the status_code field was not populated correctly in the input-metrics Kafka topic when Traces backward compatibility was enabled in NG 3.2.1. The incorrect status_code values affected the accuracy of metrics displayed in legacy Trace dashboards, particularly for failed requests. The OTEL Collector now populates the appropriate status_code for successful and failed requests, ensuring that legacy Trace dashboards display the corresponding metrics correctly.

• VUQA-7102: Updated vublock.json for the metrics status_code

Distributed Trace ID Propagation Fix​

Fixed an issue where the same transaction could generate different Trace IDs across the IBMB, FRM, and CBS services. This caused a single transaction to appear as multiple independent traces and made end-to-end transaction tracing difficult.

The Trace ID is now correctly propagated across service boundaries, allowing related operations across these services to appear as a single distributed trace while maintaining separate Span IDs for individual operations.

• VUQA-7112: Distributed Trace ID is not propagated across IBMB, FRM, and CBS

Messaging System Supporting Service Name Fix​

Fixed an issue where the service name for messaging systems, such as Kafka, was not correctly resolved and displayed as a supporting service in APM. The service name is now correctly resolved, allowing messaging systems to be identified as supporting services and represented correctly in the APM service view.

• VUQA-7141: Supporting service name for messaging system not resolved

OmniAgent Bug Fixes​

Windows OmniAgent Installation Fix​

Fixed an issue where the Windows OmniAgent installation command included the --enable-instrumentation option, causing installation to fail on Windows environments where ADI instrumentation is not supported. The Windows installation command no longer includes the --enable-instrumentation option, allowing OmniAgent to install successfully without attempting to deploy unsupported instrumentation components.

• NCS-2686: Remove --enable-instrumentation from Windows OmniAgent install command

Log Agent Bug Fixes​

AIX vuLogAgent Log Forwarding Fix​

Fixed an issue where vuLogAgent on AIX stopped forwarding logs after running for a period of time. The agent could encounter errors while processing log files and stop forwarding logs from those files, affecting continuous log collection.

The log processing and buffering behavior has been improved to prevent these failures and ensure that log events are forwarded continuously. The fix also prevents previously processed events from accumulating in the message buffer and causing Kafka message-size errors.

• NCS-2646: Fix AIX vuLogAgent forwarding issue

Upgrade / Migration Bug Fixes​

Traces Kafka Engine Tables Dropped During Upgrade​

Fixed an issue where Traces Kafka engine tables could be incorrectly dropped during the NG 2.16 to NG 3.2.1 upgrade. The migration cleanup process now tracks the Kafka engine tables created during the upgrade and excludes them from legacy-table cleanup. This prevents newly created Traces Kafka engine tables from being mistakenly removed during the migration.

• VUQA-7133: Prevent Traces Kafka engine tables from being dropped during upgrade

O11ySource Post-Job Failure During Upgrade​

Fixed an issue where the O11ySource post-job failed when upgrading from NG 3.2.1.0 to NG 3.2.1. This issue could occur when the environment had been previously upgraded from NG 2.16 to NG 3.2.1.0. The post-job now completes successfully during the NG 3.2.1.0 to NG 3.2.1 upgrade, ensuring that O11ySource configurations are processed correctly.

• VUQA-7134: Fix O11ySource post-job failure during NG 3.2.1.0 to NG 3.2.1 upgrade

Migration Job Dependency Fix​

Fixed an issue in the NG 2.16 to NG 3.2.1 migration where the create-environment-awb-records job could run before modify-hs-dictionaries completed. The migration workflow now ensures that the ClickHouse dictionary configuration is updated before create-environment-awb-records runs. This prevents the job from failing when it attempts to retrieve data through dictionaries that still contain the old database connection configuration.

• VUQA-7165: Fix dependency between modify-hs-dictionaries and create-environment-awb-records during migration

Other Platform Bug Fixes​

Post-Job View Creation Failure in ClickHouse​

Fixed an issue where post-jobs could fail to create required ClickHouse views in 2-shard, 2-replica environments. The failure caused multiple SPV views to remain uncreated due to ClickHouse JOIN and data-type configuration errors. The ClickHouse configuration has been updated to ensure the required vusmart profile settings are applied to the vusmart and vusmartmanager users. The configuration that could cause unintended nullable data types during view creation has also been removed.

Post-jobs can now successfully create the required views without encountering the reported ClickHouse errors.

• VUQA-7122: Fix View Creation Failure via Post-Jobs

Summary​

vuSmartMaps™ NG 3.2.1 delivers improvements across platform deployment, O11ySource configuration, ADI, OmniAgent, security, and Traces migration. The release introduces automated Kubernetes upgrade support, advanced O11ySource configuration, improved OmniAgent performance, and additional Kubernetes security hardening.

The release also strengthens the NG 2.16 to NG 3.2.1 Traces migration workflow with backward compatibility, rollback support, CPU capability validation, and improved trace configuration handling. In addition, multiple issues across O11ySources, APM and Tracing, data models, OmniAgent, log forwarding, and upgrade workflows have been resolved. Together, these changes improve upgrade reliability, security, configuration management, migration continuity, and overall platform stability in NG 3.2.1.

Overview​

vuSmartMaps™ NG 2.16.17 introduces license usage metering and enforcement for vuLogX log ingestion, along with enhancements to SNMP monitoring, Kafka data collection, and network device monitoring. The release also includes improvements to JBoss monitoring, documentation generation, deployment templates, dashboard data visibility, and data model permissions. These improvements help customers better manage log consumption, monitor their environments more effectively, and maintain more reliable and controlled operations.

New Features​

License Usage Metering and Enforcement for vuLogX​

Introduced license usage metering and enforcement for vuLogX log ingestion. The capability measures log ingestion volume at the point of entry, before transformation or compression, and compares consumption against the entitlement defined in the applied VuNet license.

The metering dashboard provides visibility into consumption through two views: Daily Rate Pool, which tracks usage over the trailing 24 hours, and Period Pool, which tracks cumulative usage for the license period and provides a projection of when the entitlement may be exhausted at the current usage rate.

When enabled in the license, enforcement provides two levels of protection. Daily Quota Protection temporarily pauses log ingestion when trailing 24-hour usage exceeds the configured threshold and automatically resumes ingestion when usage falls below the recovery threshold. Hard Stop stops log ingestion when cumulative usage exceeds the configured license-period threshold and requires a license upgrade before ingestion can resume.

The feature also supports automatic classification of log, metric, and trace traffic and allows custom IOStream topics to be explicitly tagged when they do not follow the standard naming convention. For more information, refer to: License Usage Metering and Enforcement — vuLogX: User Guide.

• CCBD-5759: Changes Required for License Metering and Enforcement

Enhancements​

WhatsApp Custom Channel: Custom Token Authentication​

Enhanced the WhatsApp Custom channel to support Custom Token Authentication, enabling dynamically generated authentication tokens from a separate login endpoint to be used in subsequent API requests.

• CCBD-5918: Support Custom Token Authentication in the WhatsApp Custom Channel

O11ySource: SNMP Polling Multi-Pipeline and Vendor/Model Filtering Enhancement​

Enhanced the SNMP Polling O11ySource to reduce dependency on dedicated SNMP O11ySources for vendor and model support. Vendor and model options that already have dedicated SNMP O11ySources are now excluded from the SNMP Polling configuration.

The enhancement also supports multiple pipelines with different input topics using the same output topic and prevents cleanup of base SNMP views while SNMP sources are configured through any SNMP O11ySource.

• CCBD-5599: Enhance: SNMP Polling – Multi-Pipeline Support and Exclude Vendors/Models with Dedicated SNMP O11ySources
• CCBD-5600: Support for Multi-Pipeline for a Single Data Source

O11ySource: Kafka Data Collector Configuration Enhancement​

Enhanced the Kafka Data Collector O11ySource to allow users to configure the Kafka Version and Data Format directly through the O11ySource.

Users can specify the Kafka broker/client compatibility version and select the format of messages consumed from Kafka topics, including JSON and String/Value formats. The selected configuration is automatically propagated to the generated Telegraf Kafka Data Collector configuration, eliminating the need for manual ConfigMap changes and pod restarts.

• CCBD-5732: Add Kafka Version and Data Format as User Inputs in Kafka Data Collector O11ySource

O11ySource: Fortinet Switch Device Health and Basic Information Enhancement​

Enhanced network device monitoring by adding the required Fortinet switch OIDs and support for CPU and memory health information. The enhancement also adds basic device information support for Trellix and DefensePro devices and includes the required backend changes.

• CCBD-5565: Add Device Health and Basic Information Support for Fortinet Switch

Bug Fixes​

MySQL/MariaDB O11ySource: Windows Log Path Validation Fix​

Fixed an issue where valid Windows file paths entered for the Slow Log Path and Error Log Path fields in the MySQL and MariaDB O11ySources were rejected by client-side validation. The validation pattern has been corrected to properly support Windows-style paths while continuing to support Linux paths.

• CCBD-5784: Fix Windows Log Path Validation Regex for MySQL

Firewall Requirements Documentation Generation Fix​

Fixed firewall requirement parsing during Markdown documentation generation. The documentation generator now correctly processes Source IP, Destination IP, Destination Port, Protocol, and Direction fields while preserving additional notes associated with the firewall requirements. The Office 365 setup documentation has also been updated with numbered Microsoft Cloud environments and properly formatted authentication and API endpoint links.

• CCBD-5779: Fix Markdown Documentation Generation for Firewall Requirements with Additional Notes

vutelegraf Image Tag Update​

Updated deployment templates to use the latest vutelegraf image tag, ensuring that generated deployments use the updated Telegraf image version.

• CCBD-5787: Updating Deployment Template Telegraf Image Tags to Latest

Palo Alto O11ySource Dashboard Data Fix​

Fixed an issue in the Palo Alto O11ySource dashboard where 10 panels displayed No Data instead of the expected metrics. The affected dashboard panels were corrected to display the available monitoring data.

• CCBD-5794: Palo Alto O11ySource Dashboard – 10 Panels Showing “No Data”

JBoss O11ySource CPU and Dashboard Enhancements​

Fixed the JBoss O11ySource issue that could cause high CPU utilization by optimizing the collection and processing of JBoss subsystem metrics. The updated configuration reduces unnecessary data collection while maintaining the required metric data. A new Jboss Other Insights dashboard has also been added for JCA WorkManager, RequestController, and Interface metrics. Missing ServerEnvironment fields and dashboard navigation links have been updated to improve dashboard visibility and consistency.

• CCBD-5805: JBoss – Fix NCS-2665 CPU Root Cause, Add Missing Dashboard, Complete ServerEnvironment Fields

O11ySources: Hidden Dependent Fields Reset Fix​

Fixed an issue where dependent fields that were hidden based on another configuration value could retain their previously saved values. For example, when TLS is disabled for a source, related TLS configuration fields are now cleared or stored as null instead of retaining stale values from the previous configuration.

• VUQA-7164: O11ySources – Send Rule-Hidden Fields as Null Instead of Their Stale Values

Data Model Object-Level Permission Fix​

Fixed an RBAC issue in which users with access to the Data Modelling Workspace could retrieve data from data models for which they had not been granted explicit permissions. Object-level permissions are now enforced for data model access through the Data Modelling Workspace and VED.

• VUQA-7123: Enforce Data Model Object-Level Permission on DatamodelViewSet Custom Actions

Summary​

With vuSmartMaps™ NG 2.16.17, the platform introduces license usage metering and enforcement for vuLogX and improves monitoring configuration across SNMP, Kafka, and network device environments. The release also addresses issues related to dashboard visibility, JBoss monitoring performance, documentation generation, configuration validation, deployment templates, and data model access permissions. These improvements help users manage resource usage more effectively, configure monitoring more easily, and achieve more reliable performance and access across their environments.