Private
Public Access
1
0

feat(api): add Grafana time range endpoint for scores

- Add /api/v2/server/scores/{server}/{mode} endpoint
- Support time range queries with from/to parameters
- Return data in Grafana table format for visualization
- Fix routing pattern to handle IP addresses correctly
- Add comprehensive parameter validation and error handling
This commit is contained in:
2025-07-26 12:16:55 -07:00
parent d4bf8d9e16
commit 8262b1442f
5 changed files with 618 additions and 19 deletions

View File

@@ -1,20 +1,20 @@
# DETAILED IMPLEMENTATION PLAN: Grafana Time Range API with Future Downsampling Support
## Overview
Implement a new Grafana-compatible API endpoint `/api/v2/server/scores/{server}.{mode}` that returns time series data in Grafana format with time range support and future downsampling capabilities.
Implement a new Grafana-compatible API endpoint `/api/v2/server/scores/{server}/{mode}` that returns time series data in Grafana format with time range support and future downsampling capabilities.
## API Specification
### Endpoint
- **URL**: `/api/v2/server/scores/{server}.{mode}`
- **URL**: `/api/v2/server/scores/{server}/{mode}`
- **Method**: GET
- **Path Parameters**:
- `server`: Server IP address or ID (same validation as existing API)
- `mode`: Only `json` supported initially
### Query Parameters (following Grafana conventions)
- `from`: Unix timestamp in milliseconds (required)
- `to`: Unix timestamp in milliseconds (required)
- `from`: Unix timestamp in seconds (required)
- `to`: Unix timestamp in seconds (required)
- `maxDataPoints`: Integer, default 50000, max 50000 (for future downsampling)
- `monitor`: Monitor ID, name prefix, or "*" for all (optional, same as existing)
- `interval`: Future downsampling interval like "1m", "5m", "1h" (optional, not implemented initially)
@@ -52,9 +52,11 @@ Grafana table format JSON array (more efficient than separate series):
### 1. Server Routing (`server/server.go`)
Add new route after existing scores routes:
```go
e.GET("/api/v2/server/scores/:server.:mode", srv.scoresTimeRange)
e.GET("/api/v2/server/scores/:server/:mode", srv.scoresTimeRange)
```
**Note**: Initially attempted `:server.:mode` pattern, but Echo router cannot properly parse IP addresses with dots using this pattern. Changed to `:server/:mode` to match existing API pattern and ensure compatibility with IP addresses like `23.155.40.38`.
## Key Implementation Clarifications
### Monitor Filtering Behavior
@@ -73,7 +75,7 @@ e.GET("/api/v2/server/scores/:server.:mode", srv.scoresTimeRange)
- **Minimum range**: 1 second
- **Maximum range**: 90 days
### 2. New Handler Function (`server/history.go`)
### 2. New Handler Function (`server/grafana.go`)
#### Function Signature
```go
@@ -98,7 +100,7 @@ func (srv *Server) parseTimeRangeParams(ctx context.Context, c echo.Context) (ti
return timeRangeParams{}, err
}
// Parse and validate from/to millisecond timestamps
// Parse and validate from/to second timestamps
// Validate time range (max 90 days, min 1 second)
// Parse maxDataPoints (default 50000, max 50000)
// Return extended parameters
@@ -159,7 +161,7 @@ ORDER BY ts ASC
LIMIT ?
```
### 4. Data Transformation Logic (`server/history.go`)
### 4. Data Transformation Logic (`server/grafana.go`)
#### Core Transformation Function
```go
@@ -260,7 +262,7 @@ timestampMs := logScore.Ts.Unix() * 1000
```markdown
### 7. Server Scores Time Range (v2)
**GET** `/api/v2/server/scores/{server}.{mode}`
**GET** `/api/v2/server/scores/{server}/{mode}`
Grafana-compatible time series endpoint for NTP server scoring data.
@@ -269,8 +271,8 @@ Grafana-compatible time series endpoint for NTP server scoring data.
- `mode`: Response format (`json` only)
#### Query Parameters
- `from`: Start time as Unix timestamp in milliseconds (required)
- `to`: End time as Unix timestamp in milliseconds (required)
- `from`: Start time as Unix timestamp in seconds (required)
- `to`: End time as Unix timestamp in seconds (required)
- `maxDataPoints`: Maximum data points to return (default: 50000, max: 50000)
- `monitor`: Monitor filter (ID, name prefix, or "*" for all)
@@ -319,14 +321,51 @@ Grafana table format array with one series per monitor containing all metrics as
**Recommended Grafana Data Source**: JSON API plugin (`marcusolsson-json-datasource`) - ideal for REST APIs returning table format JSON
### Phase 1: Core Implementation
- [ ] Add route in server.go
- [ ] Implement parseTimeRangeParams function
- [ ] Add LogscoresTimeRange method to ClickHouse
- [ ] Implement transformToGrafanaTableFormat function
- [ ] Add scoresTimeRange handler
- [ ] Error handling and validation (reuse existing Echo patterns)
- [ ] Cache control headers (reuse setHistoryCacheControl)
### Phase 1: Core Implementation ✅ **COMPLETED**
- [x] Add route in server.go (fixed routing pattern from `:server.:mode` to `:server/:mode`)
- [x] Implement parseTimeRangeParams function for parameter validation
- [x] Add LogscoresTimeRange method to ClickHouse with time range filtering
- [x] Implement transformToGrafanaTableFormat function with monitor grouping
- [x] Add scoresTimeRange handler with full error handling
- [x] Error handling and validation (reuse existing Echo patterns)
- [x] Cache control headers (reuse setHistoryCacheControl)
#### Phase 1 Implementation Details
**Key Components Built:**
- **Route Pattern**: `/api/v2/server/scores/:server/:mode` (matches existing API consistency)
- **Parameter Validation**: Full validation of `from`/`to` timestamps, `maxDataPoints`, time ranges
- **ClickHouse Integration**: `LogscoresTimeRange()` with time-based WHERE clauses and ASC ordering
- **Data Transformation**: Grafana table format with monitor grouping and null value handling
- **Complete Handler**: `scoresTimeRange()` with server validation, error handling, caching, and CORS
**Routing Fix**: Changed from `:server.:mode` to `:server/:mode` to resolve Echo router issue with IP addresses containing dots (e.g., `23.155.40.38`).
**Files Created/Modified in Phase 1:**
- `server/grafana.go`: Complete implementation with all structures and functions
- `timeRangeParams` struct and `parseTimeRangeParams()` function
- `transformToGrafanaTableFormat()` function with monitor grouping
- `scoresTimeRange()` handler with full error handling
- `sanitizeMonitorName()` utility function
- `server/server.go`: Added route `e.GET("/api/v2/server/scores/:server/:mode", srv.scoresTimeRange)`
- `chdb/logscores.go`: Added `LogscoresTimeRange()` method for time-based queries
**Production Testing Results** (July 25, 2025):
-**Real Data Verification**: Successfully tested with server `102.64.112.164` over 12-hour time range
-**Multiple Monitor Support**: Returns data for multiple monitors (`defra1-210hw9t`, `recentmedian`)
-**Data Quality Validation**:
- RTT conversion (microseconds → milliseconds): ✅ Working
- Timestamp conversion (seconds → milliseconds): ✅ Working
- Null value handling: ✅ Working (recentmedian has null RTT/offset as expected)
- Monitor grouping: ✅ Working (one series per monitor)
-**API Parameter Changes**: Successfully changed from milliseconds to seconds for user-friendliness
-**Volume Testing**: Handles 100+ data points per monitor efficiently
-**Error Handling**: All validation working (400 for invalid params, 404 for missing servers)
-**Performance**: Sub-second response times for 12-hour ranges
**Sample Working Request:**
```bash
curl 'http://localhost:8030/api/v2/server/scores/102.64.112.164/json?from=1753457764&to=1753500964&monitor=*'
```
### Phase 2: Testing & Polish
- [ ] Unit tests for all functions