Clapsode – High-Performance Time Series Data Layer on Azure

Next-Gen High-Performance Data Management

Clapsode provides innovative, cost-effective, and user-focused tools for complex data challenges, delivering exceptional efficiency.

Introducing Data Layer TS: Revolutionary Time Series API on Microsoft Azure. Millisecond Latency, Massive Scalability, Effortless Integration.

Explore Data Layer TS Get Started on Azure

Data Layer TS: Effortless Power for Your Time Series Data

Data Layer TS is a high-performance web API for storing and serving specific time series data (equidistant, numerical) at scale, organized within directories. Discover its key advantages.

Explore a Live Instance

See Data Layer TS in action and interact with its API documentation via Swagger UI on our live demonstration instance.

Visit datalayerts.com (Demo & API)

Unmatched Performance

Achieve millisecond latencies. A single instance processes tens of thousands of HTTP requests/sec. Batch operations handle tens of millions of data points/sec.

Massive Scalability

Easily store millions of time series, organized in directories, with millions of historical data points per series. Optimized vertical scaling prevents cluster communication overhead.

Effortless Integration

Simple HTTP API, no drivers needed. Interactive Swagger UI on every instance. Official .NET C# client library available to streamline development.

Automated & Reliable

Benefit from fully automated maintenance, ACID compliant operations, fast start-up times, and deadlock prevention. No index fragmentation worries.

Cost Efficiency

Data Layer TS’s high performance and optimized resource usage can lead to significant cost savings on consuming serverless applications and overall infrastructure.

Azure Optimized

Seamlessly deploy from Azure Marketplace, leveraging Azure’s robust infrastructure. Instances can be deallocated (pausing license fees) and benefit from optional Azure Backup for enhanced data protection.

How It Works & Key Technologies

Core Architecture Principles

Massive parallelization: Utilized for both concurrent request handling and within the execution of single, complex operations to maximize throughput and responsiveness.
Vectorized aggregations & Advanced SIMD operations: Employs AVX-512 and other SIMD instructions for rapid, parallel processing of data, significantly speeding up analytical computations.
Optimized, proprietary file structures: Custom-designed file formats ensure highly efficient storage and fast retrieval of time series data, minimizing I/O bottlenecks.
Intelligent in-memory data management: A sophisticated caching system, tailored for time series workloads, keeps frequently accessed and recent data readily available for ultra-fast access.
Vertical scaling approach: Designed to maximize the resources of a single node, this approach simplifies architecture and avoids the communication overhead often found in distributed clusters.

Technical Specifications Snapshot

Data Organization: Time series are organized within user-defined `directories`, providing a structured way to manage large collections of data.
Data Type: Specialized for equidistant numerical time series data, supporting both `SINGLE` (float) and `DOUBLE` (double) precision types.
API: A simple yet powerful RESTful HTTP API, compliant with OpenAPI v3, serves as the interface for all interactions.
Data Interaction: Utilizes `point` (individual values) and `vector` (arrays of values) based contract models. For optimal performance, especially with large datasets or batch operations, using `vector` contracts is highly recommended.
Data Formats: Supports JSON, CSV, MessagePack, and MemoryPack for flexible data exchange, allowing clients to choose the most suitable format for their needs.
Authentication: Employs RFC 7617 compliant HTTP Basic Authentication over HTTPS to ensure secure access to the API.
Maintenance: Features fully automated background jobs for essential tasks like data compaction and indexing, minimizing operational overhead.

Unlock Real-Time Insights Across Industries

Data Layer TS helps businesses solve tough time series data problems and drive innovation across different industries. It’s particularly well-suited for applications requiring rapid analysis of recent data, such as time series forecasting systems.

Advanced Weather & Climate Modeling

Challenge: Analyzing vast meteorological datasets, millions of grid points, and numerous variables for forecasting and climate research.

Solution: Data Layer TS rapidly ingests and queries massive weather datasets. Its millisecond performance enables on-the-fly aggregation and analysis, empowering faster insights and more accurate climate models.

FinTech & Trading Analytics

Challenge: Processing real-time market data, executing complex risk analysis, and efficiently backtesting trading strategies against historical data.

Solution: Data Layer TS provides ultra-low latency access to financial time series, enabling faster algorithmic trading, real-time risk assessment, and quicker, more accurate insights from market movements.

Industrial Monitoring & IIoT

Challenge: Analyzing high-resolution operational data from machinery for predictive maintenance and continuous process optimization efforts.

Solution: Data Layer TS effectively handles industrial data streams, facilitating rapid anomaly detection, improving overall operational efficiency, and enabling proactive maintenance strategies.

View Dashboard Demo Discuss Your Use Case

Key Technical Features

Data Layer TS is built with advanced technologies for outstanding performance, reliability, and developer experience.

Advanced SIMD & Vectorization

Leverages AVX-512 and other SIMD instructions for massively parallel data processing and ultra-fast aggregations, crucial for demanding analytical workloads.

Optimized In-Memory Management

Features intelligent, time-series-optimized in-memory management. Caches data from any historical point to the newest, prioritizing recent information for continuous writes and forecasting.

Efficient Storage & Disk I/O

Proprietary file structures and disk load balancing across multiple Azure data disks ensure high-throughput persistence and rapid data access, even with massive datasets.

Flexible Instance & Data Replication

Rapidly replicate entire Data Layer TS instances or copy specific time series segments. This is ideal for creating staging, development, or specialized analytical environments with ease.

Robust Azure Security & Backup

Supports Azure Trusted Launch and Encryption at Host for enhanced VM security. Data protection is ensured with integrated Azure Backup, configurable during instance creation.

Developer Tools & Libraries

Official .NET C# Client Library and a specialized C# library for DWD weather data ingestion to accelerate development. (See Documentation for NuGet links)

Transparent Pricing, Exceptional Value

Data Layer TS is on the Microsoft Azure Marketplace. Get straightforward licensing and unlock significant cost savings.

Data Layer TS License Fee: $0.33 per vCPU per hour

Access Data Layer TS directly through your Azure account. Benefit from integrated billing for the license fee alongside your Azure services.

Predictable License Costs: The Data Layer TS license fee is based on vCPU hours and is not dependent on the amount of data stored, nor on the number of data retrievals or ingestions.

Transparent Licensing: You pay the hourly license fee for Data Layer TS (per vCPU), plus standard costs for your chosen Azure VM and related Azure services (storage, network, etc.).

Scalable Azure Options: Choose the Azure Virtual Machine instance that perfectly fits your performance and budget needs. Your Azure infrastructure costs will vary based on your selection and usage.

Overall Cost Efficiency: The high performance of Data Layer TS can lead to reduced processing times and potentially lower costs for consuming serverless applications and overall Azure resource utilization.

Cost Control: Deallocate your Data Layer TS instances when not in use to pause license fees, optimizing your operational costs. Instances can be easily rescaled to match performance needs.

View on Azure Marketplace

Need enterprise licensing or custom deployment options? Contact us for a tailored solution.

Technical Documentation

Explore Data Layer TS in depth. Understand its API, data management, security, and operations to maximize its potential.

API Overview

Data Layer TS offers a comprehensive RESTful HTTP API, compliant with OpenAPI v3, for all interactions. Each deployed instance includes an interactive Swagger UI for easy API exploration and testing (e.g., at datalayerts.com).

Core Concepts:

The API is logically structured around key concepts to manage your time series data effectively:

Directories: These act as containers for organizing groups of time series.
Time Series: Individual sequences of equidistant, numerical data points. Each time series is defined by its technical metadata, including data type (SINGLE for float, DOUBLE for double precision) and temporal resolution (interval).

Directory & Series Management:

Endpoints are provided for creating, listing, updating, and deleting directories and the time series definitions within them. This allows for flexible organization and lifecycle management of your data structures.

Data Ingestion & Retrieval:

Core to the API are powerful operations for efficient data input and output. These utilize “data contracts” – specifically point for individual values and vector for arrays of values. For optimal performance, especially with large datasets or batch operations, using vector contracts is highly recommended. The API supports high-throughput scenarios, enabling rapid movement of large datasets.

User & Group Management:

Secure access to data is managed through user and group administration endpoints, allowing you to configure permissions and control data visibility.

System Information:

Retrieve instance status, version details, configuration, and operational metrics through dedicated system endpoints.

This streamlined API design, coupled with support for various data formats (JSON, CSV, MessagePack, MemoryPack via HTTP headers), facilitates straightforward integration into your applications and workflows.

Data Model & Technical Metadata

Data Organization:

Time series data is organized within directories. This hierarchical structure helps manage large numbers of distinct time series. Directories are created implicitly when the first series is defined within them. They can also have settings that influence the storage and behavior of the series they contain, such as shared creation timestamp information for storage optimization.

Time Series Structure & Technical Metadata:

Each time series within a directory is defined by its technical metadata and configuration. This information is crucial for the system’s operation and is managed via the series management API endpoints. Key metadata includes:

A unique series identifier (e.g., "sensor_temp_A001") within its parent directory.
A defined start timestamp (firstTimestamp).
A fixed interval (intervalTicks) between data points.
The data type of its numerical values (dataType: “SINGLE” or “DOUBLE”).
Configuration for creation timestamp storage (creationTimestampStorage).
Access control identifiers (readAccessId, writeAccessId) for permission management.
Automated data retention settings (retentionWindow).
Operational status fields (e.g., lastTimestamp, data volume indicators).

Storage Optimization:

Proprietary, optimized file structures are used for rapid writes and reads. Data is stored columnarly for effective compression and fast aggregations. Internal disk load balancing across multiple Azure data disks ensures performance as data volumes grow.

Authentication & Authorization

Authentication:

All API endpoints require authentication using HTTP Basic Authentication (RFC 7617) over HTTPS. Clients must include an Authorization header with Base64-encoded credentials.

Example Header: Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

Authorization (Access Control):

Access is controlled via a user and group-based permission system, managed via dedicated API endpoints. Access to data for a specific series is further controlled by readAccessId and writeAccessId attributes in the definition of each series, linking to user/group IDs.

Developer Tools & Libraries

Official .NET C# Client Library:

A .NET C# client library is provided by Clapsode to simplify integration for .NET developers.

Get it on NuGet: Clapsode.DataLayerTS.Client

C# Library for DWD Weather Data Ingestion:

A specialized C# library is available for ingesting DWD weather data.

Get it on NuGet: Clapsode.DataLayerTS.WeatherDataLoader

OpenAPI Specification & Client Generation:

The API adheres to the OpenAPI Specification (v3), allowing auto-generation of client libraries in various languages using tools like NSwag or OpenAPI Generator.

Operational Notes & Best Practices

Data Ingestion & Retrieval:

Utilize batch operations (vector contracts) for large data volumes to maximize throughput.
Ensure data aligns with the defined interval (intervalTicks) for each series.
Leverage server-side aggregations and transformation pipelines for efficient querying.

Instance Management:

Data Layer TS instances can be deallocated at any time to pause license fees, offering cost flexibility.
Rescaling options are available to adjust vCPU count and performance characteristics.

Data Formats:

Use appropriate Accept / Content-Type headers (JSON, CSV, MessagePack, MemoryPack). For performance-critical applications, binary formats like MessagePack or MemoryPack are recommended.

Monitoring & Backup:

Use system API endpoints for instance health and performance monitoring.
Leverage Azure Backup for robust data protection. Backups are optional, configured during instance creation via Azure Marketplace, and support restoration capabilities.

Maintenance:

Automated background jobs handle compaction and indexing. Auto-deletion of expired data is managed based on the retentionWindow defined for each series.

Get Support / Contact Us

About Us: The Clapsode Difference

We focus on innovation, expertise, and your success. See what makes Clapsode unique in high-performance data management.

Disruptive Innovation First

We tackle new challenges with a fresh perspective, prioritizing impactful innovation. Every Clapsode solution delivers unique, forward-thinking benefits and tangible value to our clients.

Expert In-House Engineering

Our agile team meticulously crafts solutions in-house. This expert-driven development process yields highly optimized and cohesive products, built with precision and dedicated care.

Customer-Centric Design

Your feedback shapes our development journey. This ensures our solutions are not just technologically advanced but also intuitive, user-friendly, and aligned with real-world needs.

Vertical Scaling Leadership

We champion vertical scaling with modern hardware. This approach prevents cluster communication overhead, offering simpler architectures and predictable high performance.

Transparency and Clarity

Transparency is vital for trust. Our clear API, interactive Swagger UI, detailed documentation, and predictable licensing allow for confident and informed evaluation of our solutions.

Focus on Simplicity & Power

Data Layer TS offers extreme performance through a remarkably simple HTTP API. Experience effortless power for your time series data, without complex client libraries.