Data Retention

Overview

Data Retention in vuSmartMaps allows administrators to define how long data should remain in different storage tiers: Hot, Warm, and Archived storage. This helps control storage usage, maintain system performance, and manage long-term data availability.

vuSmartMaps supports two broad storage categories:

1. Live Data Storage: This includes Hot and Warm storage. Data in these tiers remains available for queries and analysis.
2. Archived Storage: This is used for long-term retention. Archived data is not available for direct live queries. It must be restored before access.

Data retention helps administrators decide how long different types of observability data should remain available for active analysis and how long it should be kept for long-term reference.

Why This Feature Is Useful

In banking and payment environments, large volumes of data are generated from systems such as UPI, payment gateways, payment switches, middleware, and core banking applications. Without retention rules, data can continue to grow and increase storage usage. This can affect system performance and make data management difficult.

Data Retention is useful because it helps administrators:

• Keep recent and important data in fast storage for live analysis.
• Move older but still useful data to Warm storage.
• Retain historical data in Archived storage for long-term needs.
• Control storage usage across Hot, Warm, and Archived tiers.
• Define different retention periods for different data categories.
• Apply custom retention policies for specific tables or table patterns.
• Restore archived data back into original tables when needed.
• Ensure dashboards and reports can include restored data after restoration.

Example Scenario

A payment operations team uses vuSmartMaps to monitor transaction logs. Most logs are needed only for routine operational analysis, but some application logs may need longer retention for debugging.

An administrator can configure:

• A general Logs retention policy with:
  • Hot: 7 days
  • Warm: 3 days
  • Archive: 15 days
• A custom policy for a specific log table with:
  • Hot: 5 days
  • Warm: 2 days
  • Archive: 30 days
• A wildcard-based policy for application log tables starting with vlogs_app_ with:
  • Hot: 10 days
  • Warm: 4 days
  • Archive: 20 days

This allows routine logs to follow the standard policy, while important debugging or application logs are retained for a longer period.

When to Use This Feature

Use Data Retention when:

• Storage usage must be controlled.
• Different data categories need different retention periods.
• Recent data must remain available for fast live queries.
• Older data should move from Hot storage to Warm storage.
• Historical data must be retained in Archive storage.
• Archived data may need to be restored later.
• A default retention policy must be updated.
• A custom retention policy must be created for specific tables.
• A wildcard-based policy must apply to multiple related tables.
• User-defined retention policies must be edited or deleted.
• Multiple policies exist and the most specific table pattern must be applied.

Use Data Retention to control how long data stays in Hot, Warm, and Archived storage.

Comprehensive Understanding

The Data Retention page is used by administrators to view, create, edit, and manage retention policies for different types of data in vuSmartMaps. The page helps administrators control how long data should remain in Hot, Warm, and Archived storage.

Accessing the Page

1. Open the left navigation menu.
2. Go to Data Studio > Data Retention.
3. The system opens the Data Retention landing page.
4. By default, the page displays the HyperScale Data Store Retention Settings tab.
5. This tab shows all configured data retention policies.

Data Retention Policy Listing

The main part of the page is the policy listing table. This section shows all available retention policies in one place. Each row represents one retention policy. A policy defines how long data should stay in Hot, Warm, and Archived storage.

The listing helps administrators quickly understand:

1. Which data category the policy applies to.
2. Which table or table pattern is covered by the policy.
3. How many days the data will remain in Hot storage.
4. How many days the data will remain in Warm storage.
5. How many days the data will remain in Archived storage.
6. What actions can be performed on the policy.

Data Name

The Data Name shows the name of the data associated with the retention policy. This helps administrators identify the policy easily. For example, a policy may apply to logs, metrics, traces, transactions, or a custom data set.

Data Category

The Data Category shows the type of data covered by the policy. The supported categories are:

1. Traces
2. RUM
3. Logs
4. Metrics
5. Events
6. Transactions
7. Custom

This helps administrators group retention rules based on the nature of the data.

Table Name

The Table Name shows the table associated with the retention policy. For standard categories, vuSmartMaps uses predefined table prefixes. For example:

1. Traces use the vtraces prefix.
2. RUM uses the vrum prefix.
3. Logs use the vlogs prefix.
4. Metrics use the vmetrics prefix.
5. Events use the vevents prefix.
6. Transactions use the vtrans prefix.

For proper data retention, tables must include a mandatory timestamp column.

Hot Storage Retention

The Hot (Days) - Live value shows how long data remains in Hot storage. Hot storage is used for recent and frequently accessed data. This data is available for fast live searches, active analytics, and real-time operations.

Warm Storage Retention

The Warm (Days) - Live value shows how long data remains in Warm storage. Warm storage is used for data that is not accessed as frequently as Hot data, but still needs to be available for live queries. Warm data is available at a lower speed than Hot data.

Archive Storage Retention

The Archive (Days) value shows how long data remains in Archived storage. Archived data is used for long-term retention. It is not available for live queries. To use archived data again, it must be restored back to live storage.

Default Policy

The page includes a Default data retention policy. This policy applies when no user-defined policy matches a table. It works as the baseline policy for all data. The default policy can be edited, but it cannot be deleted. This ensures that vuSmartMaps always has a retention rule available.

Initially, the default retention values are:

1. Hot: 15 days
2. Warm: 5 days
3. Archive: 10 days

Actions Column

Each policy row includes an Actions column.

Administrators can use this column to perform policy-level actions:

1. Edit: Update retention values or policy details.
2. Delete: Delete a user-defined policy that is no longer required.

The Delete action is not available for the Default policy because the Default policy cannot be removed.

Create New Policy Button

The + button is available at the top-right corner of the table. Administrators use this button to create a new data retention setting. When clicked, a form opens where administrators can configure:

1. Data Category
2. Data Name
3. Tables included in the category
4. Table Name Wildcards, for custom categories
5. Hot retention period
6. Warm retention period
7. Archive retention period
8. Restored data retention period
9. Archival strategy

Data Retention Settings Modal

The page also provides a Data Retention Settings option. This opens a modal where administrators can configure the archival strategy. The available strategies are:

1. Archive Data Daily
2. Archive on Live Data Retention Period Expiry

These strategies decide how data is backed up or moved from Live storage to Archive storage.

Custom Category Options

When Custom is selected as the data category, additional options are available. Administrators can select specific tables from the HyperScale Data Store. They can also use Table Name Wildcards to apply a policy to multiple tables with matching prefixes.

For example, a wildcard prefix can be used to apply one retention policy to all application log tables that start with a specific prefix.

Overall Purpose of the Screen

Overall, the Data Retention page gives administrators one place to:

1. View all retention policies.
2. Understand how long each data type is retained.
3. Edit default or user-defined retention policies.
4. Create custom retention rules.
5. Delete user-defined policies.
6. Configure retention for specific tables or table patterns.
7. Control how data moves across Hot, Warm, and Archive storage.

Use the Default policy as the baseline and create user-defined policies only when a data category or table needs a different retention period.

Step-by-Step Instructions

Understand the Data Retention Hierarchy

Before creating or editing a policy, understand how data moves across storage tiers.

• Data starts in Hot Storage.
• After the configured Hot retention period, it moves to Warm Storage or is removed from Hot based on the Warm configuration.
• After the configured Warm retention period, it moves to Archive Storage or is deleted based on the Archive configuration.
• Data in Archive Storage is retained for the configured Archive retention period.
• Archived data can be restored back to the original tables from the data storage page.

Data Transition Schedule

Transition	Schedule
Hot to Warm	Runs at 00:30 every day.
Hot/Warm to Archive	Runs every 6 hours. The first execution time depends on when the DAO pod is started.

Configure the Archival Strategy

Administrators can configure how data moves from Live Storage to Archived Storage.

The available archival strategies are:

1. Archive Data Daily
2. Archive on Live Data Retention Period Expiry

These strategies are configured from the Data Retention Settings option.

Archive Data Daily

In this strategy, data is archived daily, starting from the previous day, also referred to as N-1.

In this strategy:

• Data in Hot and Warm storage is backed up to Archive Storage.
• A copy of the data is stored in Archive even while the data is still present in Hot or Warm storage.
• Retention for Hot, Warm, and Archive is calculated separately.

Example configuration:

Storage Tier	Retention
Hot	3 days
Warm	1 day
Archive	4 days

Example flow:

1. Data remains in Hot Storage for 3 days.
2. During this time, it is also backed up to Archive Storage.
3. After 3 days, it moves to Warm Storage for 1 day.
4. After the Warm retention period, it moves to Archive Storage for another 4 days.

Example timeline, assuming today is the 20th:

Storage Tier	Data Available
Hot Storage	17th to 19th
Warm Storage	16th
Archive Storage	12th to 19th

Total retention across tiers is:

3 + 1 + 4 = 8 days

In this strategy, Archive also stores backup data from Hot and Warm before the data moves out of Live storage.

Archive on Live Data Retention Period Expiry

In this strategy, Archive does not maintain a backup while the data is in Hot or Warm storage.

In this strategy:

• Data stays in Hot Storage for the configured Hot retention period.
• It then moves to Warm Storage for the configured Warm retention period.
• Only after the Hot and Warm retention periods expire, the data moves to Archive Storage.
• No additional backup copy is stored in Archive during the Hot or Warm period.

Example configuration:

Storage Tier	Retention
Hot	3 days
Warm	1 day
Archive	4 days

Example flow:

1. Data stays in Hot Storage for 3 days.
2. After 3 days, it moves to Warm Storage for 1 day.
3. After the Warm retention period expires, it moves to Archive Storage for 4 days.

Example timeline, assuming today is the 20th:

Storage Tier	Data Available
Hot Storage	17th to 19th
Warm Storage	16th
Archive Storage	12th to 15th

This strategy does not include a backup step. Data simply moves between Hot, Warm, and Archive based on the configured retention periods.

Configure Restored Data Retention Period

The system allows administrators to specify how long restored data should remain in the main storage tables.

This is configured through the Enter Restored Data Retention Period field.

Key behavior:

1. The default restored data retention period is 7 days.
2. Restored data is restored directly to the original tables.
3. Existing dashboards can show restored data because the data returns to the original table.
4. The system uses a specific Delete Date for restored data.
5. This prevents very old restored data from being deleted immediately.
6. Administrators can customize the restored data retention period based on business needs.

How it works:

1. When data is restored, the system calculates how long it should stay based on the configured setting.
2. The restored data is merged back into the main table.
3. The system tracks the restored data.
4. The restored data remains available until the configured retention period expires.
5. After the retention period expires, the system automatically cleans it up to save space.

note

• The restored data retention period is independent of the original archival strategy.
• If restored data is still within the standard Hot or Warm retention period, it remains in the table naturally.
• The restored data retention setting acts as a safety net for older data so that users have enough time to view it before it is removed again.

Review Existing Data Retention Policies

On the Data Retention listing page, review the configured policies.

The listing page shows:

1. Data Name
2. Data Category
3. Table Name
4. Hot retention in days
5. Warm retention in days
6. Archive retention in days
7. Actions

Use this page to understand which retention policies are already configured before creating a new one.

Edit the Default Policy

The default policy applies to all data when no user-defined policy is configured.

Initial default retention periods are:

Storage Tier	Default Retention
Hot	15 days
Warm	5 days
Archive	10 days

To edit the default policy:

1. Go to Data Studio > Data Retention.
2. Locate the Default policy row.
3. In the Actions column, click the Edit icon.
4. Modify the retention values for Hot, Warm, and Archive storage.
5. Click Save.

note

The default policy cannot be deleted.

Create a New Data Retention Setting

Use this option to create a user-defined retention policy.

1. Go to Data Studio > Data Retention.
2. Click the + button at the top-right corner of the table.
3. Complete the form.
4. Click Save to enable and apply the retention setting.

Form Configuration Steps

1. Select the Data Category.
2. Review or enter the Data Name.
3. Review or select the tables included in the category.
4. Configure the Data Retention Period for Live Data.
5. Configure the Data Retention Period for Archived Data.
6. Configure the archival strategy, if available in the form.
7. Configure the restored data retention period, if required.
8. Click Save.

Configure Data Category

Select the category of data for which the retention setting applies.

Supported categories are:

1. Traces
2. RUM
3. Logs
4. Metrics
5. Events
6. Transactions
7. Custom

Data Category and Table Name Prefixes

Data Category	Table Name Prefix
Traces	vtraces
RUM	vrum
Logs	vlogs
Metrics	vmetrics
Events	vevents
Transactions	vtrans

note

When creating a new Data Retention setting, only data category options for which retention is not already configured are available in the drop-down menu.

Configure Data Name

The system assigns default names after selecting the category. For Custom categories, administrators can provide data names.

Configure Tables Included in the Category

The system assigns table names after selecting the category. For Custom categories, table names can be selected from a dropdown that lists all distributed tables in the HyperScale Data Store. For a Custom data category, an optional field named Table Name Wildcards is available.

In this field:

• Prefixes for table names can be specified.
• Multiple items can be added using comma-separated values.
• The prefix can include numbers, hyphens, and asterisks.
• The prefix must include at least one alphabet character.

Configure Live Data Retention

Live Data includes Hot and Warm storage.

Retain HOT Data Storage Data For

1. Enter the number of days for which matching table data should remain in Hot storage.
2. Data in Hot storage is stored on high-speed disks, such as SSD.
3. Hot storage supports quick live searches and data analytics.
4. After the configured number of days, data moves to Warm storage or is deleted from
5. Hot storage based on the Warm retention configuration.

Retain WARM Data Storage Data For

1. Enter the number of days for which matching table data should remain in Warm storage.
2. Warm data is available for live queries.
3. Warm data is slower to access than Hot data.
4. After the configured number of days, data is deleted from Warm storage.

Configure Archived Data Retention

Retain Archive Storage Data For

1. Enter the number of days for which matching table data should remain in Archived storage.
2. Data in Archived storage is not available for live queries.
3. Archived data can be brought back to Live storage using the restore option on the data storage page.
4. After the configured number of days, data is deleted from Archived storage.

Manage Individual Data Retention Policies

To edit or delete a specific policy:

1. Go to the Data Retention policy listing table.
2. Locate the required policy.
3. Use the icons in the Actions column.
4. Click Edit to modify the policy.
5. Click Delete to delete the policy.

note

Deletion of the Default policy is not permitted.

Delete Multiple Policies

To delete multiple user-defined policies:

1. Select the policies using the checkboxes.
2. Click the delete icon at the top of the table.
3. Confirm the deletion.

The default policy cannot be deleted because it acts as the baseline retention policy.

Understand Priority Handling for Policies

vuSmartMaps applies retention policies based on table name matching.

Policy priority works as follows:

1. If the table name does not match any pattern in Data Retention settings, the default retention settings are used.
2. If multiple entries match the same table name or similar table prefix pattern, the policy with the longest prefix match is applied.
3. Other matching entries are ignored.
4. This ensures that the most specific policy is applied.

Example:

If the table name is:

logs-heartbeat-daily 

And the available patterns are:

logs-\* 

logs-heartbeat\*

Then the system applies:

logs-heartbeat\*

This is because it has the longest matching prefix.

Understand Edge Cases

Edge Case 1: Archive Retention Period Is Set to 0

If Archive Retention Period is set to 0, no archival functionality is applied.

Example:

Storage Tier	Retention
Hot	3 days
Warm	1 day
Archive	0 days

For Archive Data Daily:

• 4 days of backup are still maintained.
• No extra days are retained in Archive Storage.
• Data is duplicated only for the backup period.
• Once archived, it does not stay in Archive.

For Archive on Live Data Retention Period Expiry:

• Nothing is moved to Archive Storage.
• Data remains in Hot for 3 days and Warm for 1 day.
• After those retention periods expire, the data is deleted.

Edge Case 2: Warm Storage Is Not Configured

If Warm storage is not integrated into the system, any Warm retention period is ignored.The policy becomes a two-tier setup: Hot → Archive, Warm is skipped.

For Archive Data Daily:

• Data from Hot storage is directly archived.
• Since Warm is not present, only Hot data is backed up.
• Archive contains fewer days of backup than a full Hot + Warm setup.

For Archive on Live Data Retention Period Expiry:

• Data moves from Hot directly to Archive after Hot retention expires.
• Configured Warm retention is ignored.
• Data is deleted or archived based only on Hot + Archive logic.

Example:

Storage Tier	Retention
Hot	5 days
Warm	3 days
Archive	10 days

If Warm is not configured:

Strategy	Result
Archive Data Daily	kOnly 5 days of Hot data is archived instead of 8 days.
Archive on Live Data Retention Expiry	Data is deleted or archived after 5 days, not 8 days.

Edge Case 3: Archive Storage Is Not Configured

If Archive storage is not integrated, any configured Archive retention value is ignored. No data is backed up to Archive storage.

For Archive Data Daily:

• Data from Hot and Warm is not backed up.
• Archive tier is skipped.
• Data is retained only in Hot and Warm.
• Archive does not hold backup copies.
• Long-term retention guarantees may be affected.

For Archive on Live Data Retention Period Expiry:

• Once data exceeds the Hot + Warm retention period, it is permanently deleted.
• Archive is unavailable for final transition or long-term backup.
• Policies that rely on Archive for compliance or recovery do not function.

Example:

Storage Tier	Retention
Hot	5 days
Warm	3 days
Archive	10 days

If Archive storage is not configured:

Strategy	Result
Archive Data Daily	Archive tier is empty. Backup data from Hot and Warm is not stored.
Archive on Live Data Retention Expiry	Data is deleted after 8 days. Archive retention is not applied.

Review policy priority carefully. When multiple patterns match a table, the longest prefix match is applied.

What Happens After the Steps

After a retention policy is created or updated:

1. The policy appears in the Data Retention listing page.
2. The configured retention periods apply to matching tables.
3. If no user-defined policy matches a table, the default policy is applied.
4. If multiple policies match, the longest prefix match is applied.
5. Data is retained in Hot, Warm, and Archive storage according to the selected policy and archival strategy.
6. Data is moved or backed up based on the selected archival strategy.
7. Archived data is not available for live queries until it is restored.
8. Restored data returns to the original table.
9. Dashboards and reports can include restored data because it is restored into the original table.
10. Restored data remains available until the restored data retention period expires.
11. After the configured retention period expires, data is deleted from the relevant storage tier.

Configure the restored data retention period so restored historical data remains available long enough for review.

Tips / Best Practices

• Use the Default policy as the baseline for all data.
• Create user-defined policies only when a data category or table needs a different retention period.
• Use Hot storage for data that must be queried frequently.
• Use Warm storage for data that is still useful but queried less often.
• Use Archive storage for long-term historical retention.
• Use Custom category for specific tables that need special retention.
• Use Table Name Wildcards for groups of tables with similar prefixes.
• Keep wildcard prefixes specific to avoid unexpected policy matches.
• Review policy priority when multiple table prefixes may match the same table.
• Confirm that tables include a timestamp column.
• Do not rely on Archive retention if Archive storage is not configured.
• Do not rely on Warm retention if Warm storage is not configured.
• Use the restored data retention period to keep restored historical data available long enough for review.

Troubleshooting

1. Issue: A table is using the default retention period

   • Possible Cause: The table name does not match any configured Data Retention policy.
   • Solution: Create a user-defined policy for the table or configure a matching table name wildcard.

2. Issue: The expected policy is not applied

   • Possible Cause: Multiple policies may match the table name. The system applies the policy with the longest prefix match.
   • Solution: Review all matching table name patterns and confirm that the most specific prefix is configured correctly.

3. Issue: Warm retention is configured but not applied

   • Possible Cause: Warm storage may not be integrated into the system.
   • Solution: If Warm storage is not configured, Warm retention is ignored. The policy works as a Hot to Archive setup.

4. Issue: Archive retention is configured but data is not archived

   • Possible Cause: Archive storage may not be integrated into the system.
   • Solution: If Archive storage is not configured, Archive retention is ignored and no data is backed up to Archive storage.

5. Issue: Data is deleted instead of moving to Archive

   • Possible Cause: The selected archival strategy or storage configuration may result in deletion. For example, if Archive storage is not configured, data is deleted after Hot + Warm retention expires.
   • Solution: Review the archival strategy and confirm whether Archive storage is configured.

6. Issue: No archival functionality is applied

   • Possible Cause: Archive retention may be set to 0.
   • Solution: Set a non-zero Archive retention period if archival retention is required.

7. Issue: Default policy cannot be deleted

   • Possible Cause: Deletion of the Default policy is not permitted.
   • Solution: Edit the Default policy if retention values must be changed. Do not attempt to delete it.

8. Issue: Restored old data is deleted too quickly

   • Possible Cause: Thye restored data retention period may be too short.
   • Solution: Increase the value in Enter Restored Data Retention Period based on business requirements.

9. Issue: Table is not covered properly by Data Retention

   • Possible Cause: The table may not include the mandatory timestamp column.
   • Solution: Ensure the table includes the required timestamp column.

FAQs

What is Data Retention in vuSmartMaps?

Data Retention in vuSmartMaps allows administrators to define how long data should remain in different storage tiers, such as Hot, Warm, and Archived storage. This helps manage storage usage, system performance, compliance needs, and long-term data availability.

What are Hot, Warm, and Archived storage?

Hot Storage stores recent and frequently accessed data. It is used for fast live queries, active analytics, and real-time operations.

Warm Storage stores data that is accessed less often than Hot data, but is still available for live queries.

Archived Storage stores historical or rarely accessed data for long-term retention. Archived data is not available for direct live queries and must be restored before use.

What is the difference between Live Data Storage and Archived Storage?

Live Data Storage includes Hot and Warm storage. Data in these tiers remains queryable and available for analysis.

Archived Storage is used for long-term retention. Data in Archive is not directly available for live queries. It needs to be restored before it can be accessed.

What is the default data retention policy?

The default data retention policy applies to all data tables when no user-defined policy is configured. Initially, the default retention values are:

• Hot Storage: 15 days
• Warm Storage: 5 days
• Archived Storage: 10 days

The default policy can be edited, but it cannot be deleted.

What data categories are supported in Data Retention?

vuSmartMaps supports the following data categories for retention configuration:

• Traces
• RUM
• Logs
• Metrics
• Events
• Transactions
• Custom

The Custom category is used for data that does not fit into the predefined categories.

What is the purpose of the Custom data category?

The Custom data category allows administrators to configure retention for specific tables. For custom categories, administrators can provide data names and select table names from the distributed tables available in the HyperScale Data Store.

Custom categories also support Table Name Wildcards, where prefixes can be defined to apply retention policies to groups of tables with similar names.

What are the two archival strategies available in Data Retention?

vuSmartMaps supports two archival strategies:

Archive Data Daily Data is archived daily from Hot and Warm storage. A copy of the data is stored in Archive even while it is still present in Hot or Warm storage.

Archive on Live Data Retention Period Expiry Data is moved to Archive only after the configured Hot and Warm retention periods expire. No backup copy is maintained in Archive while the data is still in Hot or Warm storage.

What happens if Archive retention is set to 0?

If the Archive Retention Period is set to 0, no archival retention is applied.

For Archive Data Daily, backup may still be maintained for the Hot and Warm period, but no extra days are retained in Archive.

For Archive on Live Data Retention Period Expiry, nothing is moved to Archive. Data remains only in Hot and Warm storage for the configured period and is deleted after those periods expire.

How does vuSmartMaps decide which retention policy to apply?

If a table does not match any configured retention policy, vuSmartMaps applies the default retention policy.

If multiple policies match the same table name or table prefix pattern, the policy with the longest prefix match is applied. This ensures that the most specific policy is used.

For example, if the table name is logs-heartbeat-daily and the available patterns are logs-* and logs-heartbeat*, the system applies logs-heartbeat* because it is the more specific match.

What happens when archived data is restored?

Archived data is restored directly into the original tables. Because the data returns to the original table, existing dashboards and reports can show the restored data without extra setup.

The restored data remains available based on the configured Restored Data Retention Period. The default value is 7 days. After this period expires, the restored data is automatically cleaned up to save space.