# Open Data Hub Documentation > APIs and data ingestion documentation for the Open Data Hub. - [Open Data Hub Documentation](/index.md) ## quickstart This guide follows the path most developers take: find the dataset you need, inspect it, then read it from the API. Most Open Data Hub data is open and needs no authentication. - [Quickstart: your first request](/quickstart.md): This guide follows the path most developers take: find the dataset you need, inspect it, then read it from the API. Most Open Data Hub data is open and needs no authentication. ## domains-and-datasets The Open Data Hub provides structured access to datasets across multiple domains through REST APIs. As an API user or ingestion creator, you interact with these datasets via standardized endpoints, metadata, and response formats. This guide covers the core concepts you need to navigate and retrieve data effectively. - [Domains and datasets](/domains-and-datasets.md): The Open Data Hub provides structured access to datasets across multiple domains through REST APIs. As an API user or ingestion creator, you interact with these datasets via standardized endpoints, metadata, and response formats. This guide covers the core concepts you need to navigate and retrieve data effectively. ## licensing The Open Data Hub provides open, freely accessible data under permissive licenses, ensuring compliance with open data principles. As an API user or ingestion creator, you can rely on structured license metadata to understand the terms under which data can be used. - [Data licensing](/licensing.md): The Open Data Hub provides open, freely accessible data under permissive licenses, ensuring compliance with open data principles. As an API user or ingestion creator, you can rely on structured license metadata to understand the terms under which data can be used. ## use-data ### authentication-and-access - [Authentication and authorization](/use-data/authentication-and-access/authentication.md): To access protected data in the Open Data Hub APIs, you must authenticate using OAuth 2.0 through Keycloak and include a valid access token in your API requests. This guide explains how to obtain and use tokens, manage their lifecycle, and request access to closed datasets. All instructions are written for API users who need to consume data from the Content API or Time Series API. - [Quotas, CLI and the R package](/use-data/authentication-and-access/quotas-and-tools.md): 1. API quotas and rate limiting ### content-api - [AlpineBits integration](/use-data/content-api/alpinebits.md): The Content API supports integration with the AlpineBits standard, enabling access to tourism data through standardized endpoints. Open Data Hub implements two key datasets defined by the AlpineBits Alliance: HotelData and DestinationData. These datasets are served via separate endpoints and require specific request formatting to retrieve data successfully. - [Deprecations and datamodel migration](/use-data/content-api/deprecations.md): 1. Introduction - [Filtering and sorting (Content API)](/use-data/content-api/filtering-and-sorting.md): The Content API provides a comprehensive set of query parameters to filter, sort, and shape the data returned from its endpoints. These tools allow API users to retrieve only the data they need, reduce payload size, and organize results effectively. This guide covers the core filtering and sorting capabilities available across Content API endpoints. - [Output formats (Content API)](/use-data/content-api/output-formats.md): 1. JSON and JSON-LD response formats - [Content API reference](/use-data/content-api/reference.md): The Content API provides access to tourism-related datasets such as accommodations, activities, events, weather, and more. It exposes a set of standardized endpoints for retrieving structured data, with consistent query parameters, response formats, and metadata across all entity types. - [Content API explorer](/use-data/content-api/swagger.md): Try the Content API directly in your browser below. You can also open the full Swagger UI on the API host, which is the most reliable place to run live requests. ### geo-api - [Geo API](/use-data/geo-api/reference.md): The Geo API provides fast, map-optimized access to Open Data Hub geospatial datasets — points of interest, hiking trails, cycleways, and more — served as vector tiles for efficient rendering at any zoom level. ### time-series-api - [Filtering time series data](/use-data/time-series-api/filtering.md): The Time Series API supports flexible filtering of mobility data using the where query parameter with dot-notation operators. This enables precise retrieval of station and time series data based on attribute values, spatial relationships, and logical combinations. Unlike SQL, the filter syntax uses a functional, dot-separated format that is both human-readable and URL-safe. - [Time Series API reference](/use-data/time-series-api/reference.md): The Time Series API provides access to mobility-related time series data through a unified endpoint structure. This reference describes the API's base structure, available representations, filtering capabilities, query parameters, and response format. - [Time Series API explorer](/use-data/time-series-api/swagger.md): Try the Time Series API directly in your browser below. You can also open the full Swagger UI, which is the most reliable place to run live requests. ### transmodel-api - [Transmodel API: NeTEx and SIRI-Lite](/use-data/transmodel-api/reference.md): The Transmodel API provides standardized access to both static and real-time mobility data in compliance with European and Italian interoperability standards. It supports two key formats: NeTEx for scheduled data and SIRI-Lite for real-time updates. This API is part of the MaaS4Italy initiative and integrates with the Open Data Hub, enabling third-party applications to retrieve mobility information through the National Access Point (NAP). - [Transmodel API explorer](/use-data/transmodel-api/swagger.md): Try the Transmodel API directly in your browser below. You can also open the full Swagger UI, which is the most reliable place to run live requests. ## data-ingestion ### collector-blueprints - [API Crawler Collector](/data-ingestion/collector-blueprints/api-crawler.md): The API Crawler collector is designed to pull data from external APIs based on a declarative configuration. It allows for complex and dynamic API interactions, including multi-step calls, data transformations, and context-based processing. This collector is ideal for scenarios where data needs to be fetched from various API endpoints, potentially requiring pagination, authentication, or nested requests. - [MQTT Client Collector](/data-ingestion/collector-blueprints/mqtt-client.md): A pre-built microservice designed to easily integrate with data sources that publish information via the MQTT protocol. - [Rest Push Collector](/data-ingestion/collector-blueprints/rest-push.md): The Rest Push collector acts as a generic endpoint where external data providers can push their data directly to the Open Data Hub. This means you, as a data provider, don't need to deploy any additional software or agents on your side. Instead, you'll be provided with specific credentials and a URL, and you can simply send your data via standard HTTP POST requests. - [S3 Poller Collector](/data-ingestion/collector-blueprints/s3-poller.md): Designed for efficiently collecting data from AWS S3 buckets. This collector is ideal for scenarios where your data provider regularly uploads files to an S3 bucket, and you need to automatically retrieve these files and feed their content into the Open Data Hub pipeline. ### data-collector-from-scratch Developing a robust and reliable data collector is a critical step in any data integration pipeline. This guide will walk you through the essential principles, architectural patterns, and practical implementation details for building a data collector in Go, using the provided S3 poller example as a reference. - [Developing a Data Collector from Scratch](/data-ingestion/data-collector-from-scratch.md): Developing a robust and reliable data collector is a critical step in any data integration pipeline. This guide will walk you through the essential principles, architectural patterns, and practical implementation details for building a data collector in Go, using the provided S3 poller example as a reference. ### data-transformer-from-scratch A data transformer is the bridge between the raw data a collector produces and the standardized entities the Open Data Hub serves. It consumes raw data events from the message queue, transforms them, and pushes the result to the Open Data Hub. Where it pushes depends on the kind of data: - [Developing a Data Transformer from Scratch](/data-ingestion/data-transformer-from-scratch.md): A data transformer is the bridge between the raw data a collector produces and the standardized entities the Open Data Hub serves. It consumes raw data events from the message queue, transforms them, and pushes the result to the Open Data Hub. Where it pushes depends on the kind of data: - [Pushing to the Content API](/data-ingestion/data-transformer-from-scratch/content.md): This page covers the write side of a transformer that produces structured content (announcements, points of interest, events, and similar), using the clib package of the opendatahub-go-sdk. For the shared transformer skeleton (the SDK listener, environment variables, containerization, and local run), see Developing a Data Transformer from Scratch. - [Pushing to the Timeseries Writer (BDP)](/data-ingestion/data-transformer-from-scratch/timeseries.md): This page covers the write side of a transformer that produces time series measurements, using the go-bdp-client SDK. For the shared transformer skeleton (the SDK listener, environment variables, containerization, and the local development workflow), see Developing a Data Transformer from Scratch. ### development - [Structure:](/data-ingestion/development/intro.md): A data integration pipeline consists of at least 2 microservices: ### getting-started This guide provides a comprehensive overview of setting up your local development environment for Open Data Hub data collectors and transformers, focusing on Go-based implementations. You'll learn how to leverage Docker Compose for infrastructure, interact with essential endpoints, handle authentication for the Timeseries Writer, and get hands-on with creating and testing a collector and a transformer. - [Getting Started](/data-ingestion/getting-started.md): This guide provides a comprehensive overview of setting up your local development environment for Open Data Hub data collectors and transformers, focusing on Go-based implementations. You'll learn how to leverage Docker Compose for infrastructure, interact with essential endpoints, handle authentication for the Timeseries Writer, and get hands-on with creating and testing a collector and a transformer. ### sdks The Open Data Hub provides Software Development Kits (SDKs) to streamline the development of data integration components, such as data collectors and transformers. These SDKs abstract away common concerns like messaging, telemetry, and interaction with the Timeseries Writer and Content APIs, allowing developers to focus on the core business logic of data acquisition and transformation. - [SDKs](/data-ingestion/sdks.md): The Open Data Hub provides Software Development Kits (SDKs) to streamline the development of data integration components, such as data collectors and transformers. These SDKs abstract away common concerns like messaging, telemetry, and interaction with the Timeseries Writer and Content APIs, allowing developers to focus on the core business logic of data acquisition and transformation. ### transformer-blueprints - [Transformer Boilerplate Generator](/data-ingestion/transformer-blueprints/transformer-boilerplate.md): As of now, the Open Data Hub does not offer a wide collection of pre-configured "Transformer Blueprints" in the same way as some Data Collectors. Instead, we provide a boilerplate generator that sets up a minimal, ready-to-extend Go-based transformer service. This allows you to quickly get started with a functional transformer and then customize its core transformation logic to fit your specific data requirements. ## tools ### analytics Open Data Hub Analytics is a web tool for visualizing Open Data Hub time series data without writing code. It is useful for quickly seeing how a measurement behaves over time, comparing stations, and sharing a view with others. - [Analytics](/tools/analytics.md): Open Data Hub Analytics is a web tool for visualizing Open Data Hub time series data without writing code. It is useful for quickly seeing how a measurement behaves over time, comparing stations, and sharing a view with others. ### data-browser The Open Data Hub Data Browser is a web-based tool for exploring, filtering, and managing Open Data Hub datasets through a user-friendly interface. For API users it is a convenient way to see the exact data model and fields the Content API returns; for content editors it is also where tourism content is created and maintained. - [Data Browser](/tools/data-browser.md): The Open Data Hub Data Browser is a web-based tool for exploring, filtering, and managing Open Data Hub datasets through a user-friendly interface. For API users it is a convenient way to see the exact data model and fields the Content API returns; for content editors it is also where tourism content is created and maintained. - [Managing content in the Data Browser](/tools/data-browser/content-management.md): This guide covers the editorial workflows in the Data Browser: maintaining events, images, tags, and news articles. It is aimed at content editors and authorized users. If you only need to read data through the APIs, see the Data Browser overview instead. ### discovery Open Data Hub Discovery is a web tool for exploring and understanding the datasets in the Open Data Hub before you query them. It automatically discovers datasets from the Metadata API, infers their schema, and computes field-level statistics, so you can find the right dataset and the right fields without writing any code. - [Discovery](/tools/discovery.md): Open Data Hub Discovery is a web tool for exploring and understanding the datasets in the Open Data Hub before you query them. It automatically discovers datasets from the Metadata API, infers their schema, and computes field-level statistics, so you can find the right dataset and the right fields without writing any code. ### imageresizer 1. Image Proxy Functionality - [Imageresizer](/tools/imageresizer.md): 1. Image Proxy Functionality ### web-components The Open Data Hub provides a collection of reusable Web Components for integrating real-time data into websites and applications. These components, covering domains such as tourism, weather, and mobility, are designed to be easily embedded and configured without requiring deep integration with the underlying APIs. - [Web Components](/tools/web-components.md): The Open Data Hub provides a collection of reusable Web Components for integrating real-time data into websites and applications. These components, covering domains such as tourism, weather, and mobility, are designed to be easily embedded and configured without requiring deep integration with the underlying APIs. --- # Full Documentation Content # API Crawler Collector The API Crawler collector is designed to **pull** data from external APIs based on a declarative configuration. It allows for complex and dynamic API interactions, including multi-step calls, data transformations, and context-based processing. This collector is ideal for scenarios where data needs to be fetched from various API endpoints, potentially requiring pagination, authentication, or nested requests. This collector is ideal for scenarios where: * You need to integrate with a third-party service that provides a REST API but does not offer a push or streaming mechanism. * Data needs to be fetched on a regular, predictable schedule (e.g., every 15 minutes, once a day). * The integration requires handling complex interactions like authentication, pagination, or simple data transformations defined in a configuration file. info Implementation reference: ## Key Features[​](#key-features "Direct link to Key Features") * **Scheduled Execution**: Uses CRON expressions for flexible, automated polling of APIs. * **Pull-Based Model**: Actively fetches data from source APIs, making it independent of the source's ability to push. * **Highly Configurable Crawling**: Powered by the [go-silky](https://github.com/noi-techpark/go-silky) library, allowing for complex API interaction logic—including authentication, pagination, and dynamic requests—to be defined in a single YAML file. * **Batch and Stream Processing**: Can be configured to either wait for the entire crawl to finish and publish a single result, or to publish data items individually as they are discovered during the crawl. ## How it Works[​](#how-it-works "Direct link to How it Works") The API Crawler operates by reading a `configuration_file.yaml` configuration file, which defines the sequence of API calls to be made, how to process their responses, and how to manage the data flow. The collector then periodically executes these defined crawls based on a cron schedule. Data fetched and processed by the crawler is published to the Open Data Hub via the message queue (RabbitMQ) as raw JSON, similar to other collectors. ## Configuration[​](#configuration "Direct link to Configuration") The API Crawler's behavior is primarily driven by environment variables and the `configuration_file.yaml` configuration. ### Environment Variables[​](#environment-variables "Direct link to Environment Variables") The following environment variables are essential for configuring the API Crawler collector: | Variable Name | Description | Example Value | | ------------------------------- | ------------------------------------------------------------------------------------------------------ | ----------------------------- | | `MQ_URI` | The connection URI for the RabbitMQ instance. | `amqp://guest:guest@rabbitmq` | | `MQ_CLIENT` | A unique identifier for the data collector client connecting to RabbitMQ. | `dc-api-crawler-client` | | `MQ_EXCHANGE` | The RabbitMQ exchange to which the collected data will be published. | `ingress` | | `LOG_LEVEL` | The logging level for the collector. | `DEBUG` | | `PROVIDER` | The unique identifier for the data provider and dataset. This value is included in the published data. | `myprovider/mydataset` | | `CRON` | The cron schedule string defining how often the crawler should run, including seconds. | `0 * * * * *` | | `CONFIG_PATH` | The file path to the API crawler's configuration file. | `configuration_file.yaml` | | `SERVICE_NAME` | The name of the service for telemetry purposes. | `dc-api-crawler-myservice` | | `TELEMETRY_TRACE_GRPC_ENDPOINT` | The gRPC endpoint for sending telemetry traces. | `localhost:4317` | ### `configuration_file.yaml` Configuration[​](#configuration_fileyaml-configuration "Direct link to configuration_fileyaml-configuration") The `CONFIG_PATH` environment variable points to the `configuration_file.yaml` file, which is the core of the API Crawler's functionality. This YAML file uses the `go-silky` library's declarative syntax to define: * **API requests**: URLs, HTTP methods, headers, and body content. * **Authentication**: How to authenticate with the target APIs. * **Data extraction and transformation**: Using `jq` for JSON parsing and transformation. * **Flow control**: Using `foreach` loops for iterating over collections or paginated results. * **Context management**: How data from different steps is combined and made available to subsequent steps. For detailed information on configuring the `configuration_file.yaml`, please refer to the [go-silky package documentation](https://github.com/noi-techpark/go-silky). The documentation covers the schema for `request` and `foreach` steps, authentication methods, `jq` and Go template usage, and context management. The `go-silky` [configuration builder IDE](https://github.com/noi-techpark/go-silky/releases) is highly recommended for developing and debugging your `configuration_file.yaml` files, allowing you to execute and inspect the configuration in real-time. --- # MQTT Client Collector A pre-built microservice designed to easily integrate with data sources that publish information via the MQTT protocol. This blueprint is perfect for scenarios where your data provider or IoT devices send real-time updates to an MQTT broker. Instead of developing a custom collector from scratch, you can deploy and configure this reusable component to automatically receive these messages and forward them into the Open Data Hub pipeline. info Implementation reference: ## What is the MQTT Listener?[​](#what-is-the-mqtt-listener "Direct link to What is the MQTT Listener?") The MQTT Listener is a specialized data collector that acts as a subscriber to an MQTT broker. It listens for messages on a specified MQTT topic, captures the raw payload of these messages, and then publishes this raw data to an internal Open Data Hub message queue (RabbitMQ). From there, the data can be picked up by a transformer for further processing and standardization. This collector operates on a "listen" model, meaning it continuously waits for new messages on the configured MQTT topic. ### Key Features[​](#key-features "Direct link to Key Features") * **MQTT Subscription**: Connects to an MQTT broker and subscribes to a defined topic. * **Real-time Data Capture**: Captures message ID, topic, and payload from incoming MQTT messages. * **Raw Data Forwarding**: Publishes the raw MQTT message content directly to an Open Data Hub message queue (RabbitMQ). * **Configurable Credentials**: Supports MQTT username/password authentication. * **Flexible Routing**: Allows configuration of RabbitMQ exchange and routing keys. ## How to Configure the MQTT Listener[​](#how-to-configure-the-mqtt-listener "Direct link to How to Configure the MQTT Listener") Configuring the MQTT Listener is primarily done through environment variables. This allows for easy deployment and management across different environments (development, testing, production) without modifying the code. You will typically configure these variables in your deployment setup (e.g., in a `.env` file for local development, or in Helm charts for Kubernetes deployments). ### Common Environment Variables[​](#common-environment-variables "Direct link to Common Environment Variables") These variables are standard across many Open Data Hub collectors and define logging and basic messaging behavior. | Variable Name | Description | Example Value | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | `LOG_LEVEL` | The minimum logging level (e.g., `DEBUG`, `INFO`, `WARN`, `ERROR`). | `INFO` | | `MQ_EXCHANGE` | The RabbitMQ exchange where raw data messages will be published. | `ingress` | | `MQ_CLIENT` | A unique identifier for this collector client, used by RabbitMQ for connection naming. | `dc-mqtt-client-dev` | | `PROVIDER` | A unique identifier for the data source this collector is handling (e.g., `source-name/dataset-name`). This will be part of the raw data metadata. | `test/mqtt` | ### MQTT-Specific Environment Variables[​](#mqtt-specific-environment-variables "Direct link to MQTT-Specific Environment Variables") These variables are unique to the MQTT Listener and define its connection to your MQTT broker. | Variable Name | Description | Example Value | | --------------- | ------------------------------------------------------------------------------------------------------- | ----------------------------- | | `MQTT_URI` | The URI of your MQTT broker (e.g., `tcp://broker.example.com:1883` or `ssl://broker.example.com:8883`). | `xsona-broker.axians.it:8883` | | `MQTT_USER` | The username for connecting to the MQTT broker. | `my_mqtt_user` | | `MQTT_PASS` | The password for connecting to the MQTT broker. | `my_mqtt_password` | | `MQTT_CLIENTID` | A unique client ID for this MQTT subscriber. | `airQuinoLinearizationDev` | | `MQTT_TOPIC` | The MQTT topic to subscribe to for incoming messages. | `AirQuino/RawData` | --- # Rest Push Collector The **Rest Push** collector acts as a generic endpoint where external data providers can **push** their data directly to the Open Data Hub. This means you, as a data provider, don't need to deploy any additional software or agents on your side. Instead, you'll be provided with specific credentials and a URL, and you can simply send your data via standard HTTP POST requests. This collector is ideal for scenarios where: * You have an existing system that can make HTTP requests. * You want a straightforward way to send data without managing complex data pipelines or security configurations yourself. * Your data naturally fits into a "push" model (e.g., event streams, periodic reports). info Implementation reference: ## Key Features[​](#key-features "Direct link to Key Features") * **HTTP POST Endpoint**: A dedicated URL to receive incoming data. * **Secure Authentication**: Leverages Keycloak for OAuth2-based authentication using client credentials. * **Fine-grained Authorization**: Employs Keycloak UMA (User-Managed Access) to control access to specific data streams (`provider`/`dataset` paths). * **Simple Data Ingestion**: Data is sent directly in the request body, supporting various content types. ## How to Configure and Use It (for Data Providers)[​](#how-to-configure-and-use-it-for-data-providers "Direct link to How to Configure and Use It (for Data Providers)") To utilize the Inbound REST API collector, you will primarily interact with the Open Data Hub through API calls. The Open Data Hub team will handle the underlying deployment and Keycloak setup, providing you with the necessary access details. ### 1. Obtain Credentials and Endpoint URL[​](#1-obtain-credentials-and-endpoint-url "Direct link to 1. Obtain Credentials and Endpoint URL") Before you can push data, the Open Data Hub team will provide you with: * A **Keycloak `client_id`** and `client_secret`. * The **Keycloak authentication URL** (token endpoint). * Your specific **``** and **``** identifiers, which define the path for your data. - Production Environment - Testing Environment For the **Production** environment, the base URL for the Inbound REST API collector is: `https://push.api.opendatahub.com` For the **Testing** environment, the base URL for the Inbound REST API collector is: `https://push.api.dev.opendatahub.testingmachine.eu` ### 2. Get an Access Token[​](#2-get-an-access-token "Direct link to 2. Get an Access Token") All data pushes require an OAuth2 access token for authentication. You'll use the `client_credentials` grant type with your provided `client_id` and `client_secret`. Here's how you can obtain an access token using `curl`: ``` # Replace with your actual Keycloak URL, Realm, Client ID, and Client Secret KEYCLOAK_TOKEN_URL="YOUR_KEYCLOAK_AUTH_URL/realms/YOUR_KEYCLOAK_REALM/protocol/openid-connect/token" CLIENT_ID="your-client-id" CLIENT_SECRET="your-client-secret" ACCESS_TOKEN=$(curl -s -X POST "${KEYCLOAK_TOKEN_URL}" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=${CLIENT_ID}" \ -d "client_secret=${CLIENT_SECRET}" | jq -r '.access_token') echo "Obtained Access Token: ${ACCESS_TOKEN}" ``` * **`YOUR_KEYCLOAK_AUTH_URL`**: The base URL of your Keycloak instance (e.g., `https://auth.opendatahub.example.com`). * **`YOUR_KEYCLOAK_REALM`**: The specific realm configured for your Open Data Hub environment. * **`your-client-id`**: The `client_id` provided by the Open Data Hub team. * **`your-client-secret`**: The `client_secret` associated with your `client_id`. * `jq -r '.access_token'`: This uses `jq` to parse the JSON response and extract only the access token. Make sure `jq` is installed on your system (`sudo apt-get install jq` or `brew install jq`). The `ACCESS_TOKEN` variable will now hold your valid token, which typically expires after a short period (e.g., 5 minutes). You'll need to re-fetch a new token when it expires. ### 3. Push Your Data[​](#3-push-your-data "Direct link to 3. Push Your Data") Once you have an `ACCESS_TOKEN`, you can push your data to the collector's endpoint. The endpoint format is: `YOUR_COLLECTOR_BASE_URL/push//` * **`YOUR_COLLECTOR_BASE_URL`**: The base URL for the Inbound REST API collector. * **``**: Your unique identifier as a data provider. * **``**: The identifier for the specific dataset you are pushing. Here's a `curl` example for pushing JSON data: ``` # Replace with your actual collector base URL, provider, dataset, and your JSON payload COLLECTOR_URL="YOUR_COLLECTOR_BASE_URL/push/your-provider-id/your-dataset-id" DATA_PAYLOAD='{"timestamp": "2025-07-24T12:00:00Z", "value": 123.45, "unit": "°C"}' curl -s -X POST "${COLLECTOR_URL}" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${ACCESS_TOKEN}" \ -d "${DATA_PAYLOAD}" ``` * **`Content-Type` header**: Set this to match the format of your `DATA_PAYLOAD` (e.g., `application/json`, `text/csv`, etc.). * **`Authorization: Bearer ${ACCESS_TOKEN}`**: This header is crucial for authenticating your request. Upon successful push, you will receive a `200 OK` response with a message confirming data acceptance and a unique ID for the pushed message. ### Understanding Authorization (Keycloak UMA)[​](#understanding-authorization-keycloak-uma "Direct link to Understanding Authorization (Keycloak UMA)") The collector uses Keycloak's User-Managed Access (UMA) for fine-grained authorization. This means that even with a valid access token, your client needs specific *permissions* to push data to a particular `//` path. The Open Data Hub team will configure these permissions in Keycloak for you. They will create a "resource" representing your data stream (e.g., `/provider-id/dataset-id`), define a "policy" that specifies who (your client) can access it, and then link them via a "permission". This ensures that only authorized clients can send data to designated endpoints. ### Other Endpoints[​](#other-endpoints "Direct link to Other Endpoints") * **`/health` (GET)**: A simple health check endpoint to verify if the collector service is running. * **`/apispec` (GET)**: Provides the OpenAPI 3.0 specification (YAML format) for the collector's API, which can be useful for developers integrating with the service. --- # S3 Poller Collector Designed for efficiently collecting data from AWS S3 buckets. This collector is ideal for scenarios where your data provider regularly uploads files to an S3 bucket, and you need to automatically retrieve these files and feed their content into the Open Data Hub pipeline. info Implementation reference: ## What is the S3 Poller?[​](#what-is-the-s3-poller "Direct link to What is the S3 Poller?") The S3 Poller is a specialized data collector that periodically checks a configured AWS S3 bucket for a specific file. On a predefined schedule (using a cron expression), it downloads the content of this file and then publishes its raw body to an internal Open Data Hub message queue (RabbitMQ). From there, the data can be picked up by a transformer for further processing and standardization. This collector operates on a "pull" model, actively polling the S3 bucket at regular intervals. ### Key Features[​](#key-features "Direct link to Key Features") * **Scheduled Polling**: Configurable cron schedule to define how often the S3 bucket is checked. * **AWS S3 Integration**: Connects to AWS S3 using provided credentials and region. * **Specific File Retrieval**: Targets a single file within a specified S3 bucket. * **Raw Data Forwarding**: Publishes the entire content of the S3 file as raw data to an Open Data Hub message queue. * **Binary/Text Handling**: Can be configured to treat the S3 file content as raw binary data (e.g., for images, compressed files) or as a string (e.g., for JSON, CSV). ## How to Configure the S3 Poller[​](#how-to-configure-the-s3-poller "Direct link to How to Configure the S3 Poller") Configuring the S3 Poller is straightforward, primarily done through environment variables. This approach ensures flexibility and ease of deployment across different environments (development, testing, production) without requiring code changes. You will typically configure these variables in your deployment setup (e.g., in a `.env` file for local development, or in Helm charts for Kubernetes deployments). ### Common Environment Variables[​](#common-environment-variables "Direct link to Common Environment Variables") These variables are standard across many Open Data Hub collectors and define logging and basic messaging behavior. | Variable Name | Description | Example Value | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | `MQ_URI` | The RabbitMQ connection URI. | `amqp://guest:guest@rabbitmq` | | `MQ_CLIENT` | A unique identifier for this collector client, used by RabbitMQ for connection naming. | `dc-meteorology-bz-forecast` | | `MQ_EXCHANGE` | The RabbitMQ exchange where raw data messages will be published. | `ingress` | | `LOGLEVEL` | The minimum logging level (e.g., `DEBUG`, `INFO`, `WARN`, `ERROR`). | `DEBUG` | | `PROVIDER` | A unique identifier for the data source this collector is handling (e.g., `source-name/dataset-name`). This will be part of the raw data metadata. | `s3-poller/meteorology-bz-forecast` | ### S3 Poller-Specific Environment Variables[​](#s3-poller-specific-environment-variables "Direct link to S3 Poller-Specific Environment Variables") These variables are unique to the S3 Poller and define its connection to your AWS S3 bucket and the polling schedule. | Variable Name | Description | Example Value | | ----------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------- | | `CRON` | The cron schedule for polling the S3 bucket. Supports seconds precision. | `"0/10 * * * * *"` (every 10 seconds) | | `AWS_REGION` | The AWS region where your S3 bucket is located. | `eu-west-1` | | `AWS_S3_FILE_NAME` | The exact name (key) of the file to be downloaded from S3. | `SMOS_MCPL-WX_EXP_SIAG.JSON` | | `AWS_S3_BUCKET_NAME` | The name of the S3 bucket to poll. | `dc-meteorology-province-forecast` | | `AWS_ACCESS_KEY_ID` | Your AWS Access Key ID. | `AKIA...` | | `AWS_ACCESS_SECRET_KEY` | Your AWS Secret Access Key. | `yn9v...` | | `RAW_BINARY` | If `true`, the S3 file content is treated as raw binary data. If `false` (default), it's converted to a string. | `false` | --- # Developing a Data Collector from Scratch Developing a robust and reliable data collector is a critical step in any data integration pipeline. This guide will walk you through the essential principles, architectural patterns, and practical implementation details for building a data collector in Go, using the provided S3 poller example as a reference. ## 1. Understanding the Role of a Data Collector[​](#1-understanding-the-role-of-a-data-collector "Direct link to 1. Understanding the Role of a Data Collector") A data collector is a specialized microservice responsible for: * **Source Interaction**: Connecting to an external data source (e.g., an API, a database, a file system like S3, an FTP server). * **Data Retrieval**: Fetching raw data from the source. * **Raw Data Publication**: Publishing the retrieved raw data to a message queue, typically for subsequent processing by a transformer. * **Scheduling/Triggering**: Operating on a schedule (e.g., cron, polling interval) or in response to external events. The primary goal of a data collector is to reliably acquire data and make it available in a standardized raw format for the next stage of the pipeline. Typical **Pull-Data-Collector** flow (the collector actively gets data from the provider) ![pull collector flow](data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPCEtLSBEbyBub3QgZWRpdCB0aGlzIGZpbGUgd2l0aCBlZGl0b3JzIG90aGVyIHRoYW4gZHJhdy5pbyAtLT4KPCFET0NUWVBFIHN2ZyBQVUJMSUMgIi0vL1czQy8vRFREIFNWRyAxLjEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvR3JhcGhpY3MvU1ZHLzEuMS9EVEQvc3ZnMTEuZHRkIj4KPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJiYWNrZ3JvdW5kOiAjZmZmZmZmOyBiYWNrZ3JvdW5kLWNvbG9yOiBsaWdodC1kYXJrKCNmZmZmZmYsIHZhcigtLWdlLWRhcmstY29sb3IsICMxMjEyMTIpKTsgY29sb3Itc2NoZW1lOiBsaWdodCBkYXJrOyIgeG1sbnM6eGxpbms9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGxpbmsiIHZlcnNpb249IjEuMSIgd2lkdGg9IjgyMXB4IiBoZWlnaHQ9IjYxcHgiIHZpZXdCb3g9Ii0wLjUgLTAuNSA4MjEgNjEiIGNvbnRlbnQ9IiZsdDtteGZpbGUgaG9zdD0mcXVvdDthcHAuZGlhZ3JhbXMubmV0JnF1b3Q7IGFnZW50PSZxdW90O01vemlsbGEvNS4wIChYMTE7IExpbnV4IHg4Nl82NDsgcnY6MTM2LjApIEdlY2tvLzIwMTAwMTAxIEZpcmVmb3gvMTM2LjAmcXVvdDsgdmVyc2lvbj0mcXVvdDsyOC4wLjYmcXVvdDsgc2NhbGU9JnF1b3Q7MSZxdW90OyBib3JkZXI9JnF1b3Q7MCZxdW90OyZndDsmI3hBOyAgJmx0O2RpYWdyYW0gbmFtZT0mcXVvdDtQYWdlLTEmcXVvdDsgaWQ9JnF1b3Q7bE1jNmxOUExMWmR6ZU42ZGtmWmwmcXVvdDsmZ3Q7JiN4QTsgICAgJmx0O214R3JhcGhNb2RlbCBkeD0mcXVvdDsxNDI2JnF1b3Q7IGR5PSZxdW90Ozc5NyZxdW90OyBncmlkPSZxdW90OzEmcXVvdDsgZ3JpZFNpemU9JnF1b3Q7MTAmcXVvdDsgZ3VpZGVzPSZxdW90OzEmcXVvdDsgdG9vbHRpcHM9JnF1b3Q7MSZxdW90OyBjb25uZWN0PSZxdW90OzEmcXVvdDsgYXJyb3dzPSZxdW90OzEmcXVvdDsgZm9sZD0mcXVvdDsxJnF1b3Q7IHBhZ2U9JnF1b3Q7MSZxdW90OyBwYWdlU2NhbGU9JnF1b3Q7MSZxdW90OyBwYWdlV2lkdGg9JnF1b3Q7ODUwJnF1b3Q7IHBhZ2VIZWlnaHQ9JnF1b3Q7MTEwMCZxdW90OyBtYXRoPSZxdW90OzAmcXVvdDsgc2hhZG93PSZxdW90OzAmcXVvdDsmZ3Q7JiN4QTsgICAgICAmbHQ7cm9vdCZndDsmI3hBOyAgICAgICAgJmx0O214Q2VsbCBpZD0mcXVvdDswJnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDtteENlbGwgaWQ9JnF1b3Q7MSZxdW90OyBwYXJlbnQ9JnF1b3Q7MCZxdW90OyAvJmd0OyYjeEE7ICAgICAgICAmbHQ7bXhDZWxsIGlkPSZxdW90O3N0YXJ0JnF1b3Q7IHZhbHVlPSZxdW90O1RyaWdnZXImYW1wOyN4YTsoQ3Jvbi9FdmVudCkmcXVvdDsgc3R5bGU9JnF1b3Q7ZWxsaXBzZTtmaWxsQ29sb3I9I2RhZThmYztzdHJva2VDb2xvcj0jNmM4ZWJmO2ZvbnRTaXplPTEyOyZxdW90OyBwYXJlbnQ9JnF1b3Q7MSZxdW90OyB2ZXJ0ZXg9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSB4PSZxdW90OzIwJnF1b3Q7IHk9JnF1b3Q7MTQwJnF1b3Q7IHdpZHRoPSZxdW90OzEyMCZxdW90OyBoZWlnaHQ9JnF1b3Q7NjAmcXVvdDsgYXM9JnF1b3Q7Z2VvbWV0cnkmcXVvdDsgLyZndDsmI3hBOyAgICAgICAgJmx0Oy9teENlbGwmZ3Q7JiN4QTsgICAgICAgICZsdDtteENlbGwgaWQ9JnF1b3Q7c291cmNlJnF1b3Q7IHZhbHVlPSZxdW90O1NvdXJjZSBJbnRlcmFjdGlvbiZhbXA7I3hhOyhBUEkgLyBTMyAvIEZUUCkmcXVvdDsgc3R5bGU9JnF1b3Q7c2hhcGU9cmVjdGFuZ2xlO3JvdW5kZWQ9MTtmaWxsQ29sb3I9I2Q1ZThkNDtzdHJva2VDb2xvcj0jODJiMzY2O2ZvbnRTaXplPTEyOyZxdW90OyBwYXJlbnQ9JnF1b3Q7MSZxdW90OyB2ZXJ0ZXg9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSB4PSZxdW90OzE2MCZxdW90OyB5PSZxdW90OzE0MCZxdW90OyB3aWR0aD0mcXVvdDsxNjAmcXVvdDsgaGVpZ2h0PSZxdW90OzYwJnF1b3Q7IGFzPSZxdW90O2dlb21ldHJ5JnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDsvbXhDZWxsJmd0OyYjeEE7ICAgICAgICAmbHQ7bXhDZWxsIGlkPSZxdW90O2ZldGNoJnF1b3Q7IHZhbHVlPSZxdW90O0ZldGNoIERhdGEmcXVvdDsgc3R5bGU9JnF1b3Q7c2hhcGU9cmVjdGFuZ2xlO3JvdW5kZWQ9MTtmaWxsQ29sb3I9I2ZmZjJjYztzdHJva2VDb2xvcj0jZDZiNjU2O2ZvbnRTaXplPTEyOyZxdW90OyBwYXJlbnQ9JnF1b3Q7MSZxdW90OyB2ZXJ0ZXg9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSB4PSZxdW90OzM0MCZxdW90OyB5PSZxdW90OzE0MCZxdW90OyB3aWR0aD0mcXVvdDsxMjAmcXVvdDsgaGVpZ2h0PSZxdW90OzYwJnF1b3Q7IGFzPSZxdW90O2dlb21ldHJ5JnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDsvbXhDZWxsJmd0OyYjeEE7ICAgICAgICAmbHQ7bXhDZWxsIGlkPSZxdW90O3ZhbGlkYXRlJnF1b3Q7IHZhbHVlPSZxdW90O1ZhbGlkYXRlIC8gUHJlcGFyZSBSYXcgRGF0YSZxdW90OyBzdHlsZT0mcXVvdDtzaGFwZT1yZWN0YW5nbGU7cm91bmRlZD0xO2ZpbGxDb2xvcj0jZjhjZWNjO3N0cm9rZUNvbG9yPSNiODU0NTA7Zm9udFNpemU9MTI7JnF1b3Q7IHBhcmVudD0mcXVvdDsxJnF1b3Q7IHZlcnRleD0mcXVvdDsxJnF1b3Q7Jmd0OyYjeEE7ICAgICAgICAgICZsdDtteEdlb21ldHJ5IHg9JnF1b3Q7NDgwJnF1b3Q7IHk9JnF1b3Q7MTQwJnF1b3Q7IHdpZHRoPSZxdW90OzE4MCZxdW90OyBoZWlnaHQ9JnF1b3Q7NjAmcXVvdDsgYXM9JnF1b3Q7Z2VvbWV0cnkmcXVvdDsgLyZndDsmI3hBOyAgICAgICAgJmx0Oy9teENlbGwmZ3Q7JiN4QTsgICAgICAgICZsdDtteENlbGwgaWQ9JnF1b3Q7cHVibGlzaCZxdW90OyB2YWx1ZT0mcXVvdDtQdWJsaXNoIHRvIE1RJmFtcDsjeGE7KFJhYmJpdE1RKSZxdW90OyBzdHlsZT0mcXVvdDtzaGFwZT1jbG91ZDtmaWxsQ29sb3I9I2UxZDVlNztzdHJva2VDb2xvcj0jOTY3M2E2O2ZvbnRTaXplPTEyOyZxdW90OyBwYXJlbnQ9JnF1b3Q7MSZxdW90OyB2ZXJ0ZXg9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSB4PSZxdW90OzY4MCZxdW90OyB5PSZxdW90OzE0MCZxdW90OyB3aWR0aD0mcXVvdDsxNjAmcXVvdDsgaGVpZ2h0PSZxdW90OzYwJnF1b3Q7IGFzPSZxdW90O2dlb21ldHJ5JnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDsvbXhDZWxsJmd0OyYjeEE7ICAgICAgICAmbHQ7bXhDZWxsIGlkPSZxdW90O2ExJnF1b3Q7IHN0eWxlPSZxdW90O2VuZEFycm93PWJsb2NrO3N0cm9rZVdpZHRoPTI7JnF1b3Q7IHBhcmVudD0mcXVvdDsxJnF1b3Q7IHNvdXJjZT0mcXVvdDtzdGFydCZxdW90OyB0YXJnZXQ9JnF1b3Q7c291cmNlJnF1b3Q7IGVkZ2U9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSByZWxhdGl2ZT0mcXVvdDsxJnF1b3Q7IGFzPSZxdW90O2dlb21ldHJ5JnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDsvbXhDZWxsJmd0OyYjeEE7ICAgICAgICAmbHQ7bXhDZWxsIGlkPSZxdW90O2EyJnF1b3Q7IHN0eWxlPSZxdW90O2VuZEFycm93PWJsb2NrO3N0cm9rZVdpZHRoPTI7JnF1b3Q7IHBhcmVudD0mcXVvdDsxJnF1b3Q7IHNvdXJjZT0mcXVvdDtzb3VyY2UmcXVvdDsgdGFyZ2V0PSZxdW90O2ZldGNoJnF1b3Q7IGVkZ2U9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSByZWxhdGl2ZT0mcXVvdDsxJnF1b3Q7IGFzPSZxdW90O2dlb21ldHJ5JnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDsvbXhDZWxsJmd0OyYjeEE7ICAgICAgICAmbHQ7bXhDZWxsIGlkPSZxdW90O2EzJnF1b3Q7IHN0eWxlPSZxdW90O2VuZEFycm93PWJsb2NrO3N0cm9rZVdpZHRoPTI7JnF1b3Q7IHBhcmVudD0mcXVvdDsxJnF1b3Q7IHNvdXJjZT0mcXVvdDtmZXRjaCZxdW90OyB0YXJnZXQ9JnF1b3Q7dmFsaWRhdGUmcXVvdDsgZWRnZT0mcXVvdDsxJnF1b3Q7Jmd0OyYjeEE7ICAgICAgICAgICZsdDtteEdlb21ldHJ5IHJlbGF0aXZlPSZxdW90OzEmcXVvdDsgYXM9JnF1b3Q7Z2VvbWV0cnkmcXVvdDsgLyZndDsmI3hBOyAgICAgICAgJmx0Oy9teENlbGwmZ3Q7JiN4QTsgICAgICAgICZsdDtteENlbGwgaWQ9JnF1b3Q7YTQmcXVvdDsgc3R5bGU9JnF1b3Q7ZW5kQXJyb3c9YmxvY2s7c3Ryb2tlV2lkdGg9MjsmcXVvdDsgcGFyZW50PSZxdW90OzEmcXVvdDsgc291cmNlPSZxdW90O3ZhbGlkYXRlJnF1b3Q7IHRhcmdldD0mcXVvdDtwdWJsaXNoJnF1b3Q7IGVkZ2U9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSByZWxhdGl2ZT0mcXVvdDsxJnF1b3Q7IGFzPSZxdW90O2dlb21ldHJ5JnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDsvbXhDZWxsJmd0OyYjeEE7ICAgICAgJmx0Oy9yb290Jmd0OyYjeEE7ICAgICZsdDsvbXhHcmFwaE1vZGVsJmd0OyYjeEE7ICAmbHQ7L2RpYWdyYW0mZ3Q7JiN4QTsmbHQ7L214ZmlsZSZndDsmI3hBOyI+PGRlZnMvPjxyZWN0IGZpbGw9IiNmZmZmZmYiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKCNmZmZmZmYsIHZhcigtLWdlLWRhcmstY29sb3IsICMxMjEyMTIpKTsiIHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHg9IjAiIHk9IjAiLz48Zz48ZyBkYXRhLWNlbGwtaWQ9IjAiPjxnIGRhdGEtY2VsbC1pZD0iMSI+PGcgZGF0YS1jZWxsLWlkPSJzdGFydCI+PGc+PGVsbGlwc2UgY3g9IjYwIiBjeT0iMzAiIHJ4PSI2MCIgcnk9IjMwIiBmaWxsPSIjZGFlOGZjIiBzdHlsZT0iZmlsbDogbGlnaHQtZGFyayhyZ2IoMjE4LCAyMzIsIDI1MiksIHJnYigyOSwgNDEsIDU5KSk7IHN0cm9rZTogbGlnaHQtZGFyayhyZ2IoMTA4LCAxNDIsIDE5MSksIHJnYig5MiwgMTIxLCAxNjMpKTsiIHN0cm9rZT0iIzZjOGViZiIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjwvZz48Zz48ZyBmaWxsPSIjMDAwMDAwIiBzdHlsZT0iZmlsbDogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEycHgiPjx0ZXh0IHg9IjYwIiB5PSIyOCI+VHJpZ2dlcjwvdGV4dD48dGV4dCB4PSI2MCIgeT0iNDIiPihDcm9uL0V2ZW50KTwvdGV4dD48L2c+PC9nPjwvZz48ZyBkYXRhLWNlbGwtaWQ9InNvdXJjZSI+PGc+PHJlY3QgeD0iMTQwIiB5PSIwIiB3aWR0aD0iMTYwIiBoZWlnaHQ9IjYwIiByeD0iOSIgcnk9IjkiIGZpbGw9IiNkNWU4ZDQiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigyMTMsIDIzMiwgMjEyKSwgcmdiKDMxLCA0NywgMzApKTsgc3Ryb2tlOiBsaWdodC1kYXJrKHJnYigxMzAsIDE3OSwgMTAyKSwgcmdiKDY4LCAxMTAsIDQ0KSk7IiBzdHJva2U9IiM4MmIzNjYiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48L2c+PGc+PGcgZmlsbD0iIzAwMDAwMCIgc3R5bGU9ImZpbGw6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyIgZm9udC1mYW1pbHk9IkhlbHZldGljYSIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMnB4Ij48dGV4dCB4PSIyMjAiIHk9IjI4Ij5Tb3VyY2UgSW50ZXJhY3Rpb248L3RleHQ+PHRleHQgeD0iMjIwIiB5PSI0MiI+KEFQSSAvIFMzIC8gRlRQKTwvdGV4dD48L2c+PC9nPjwvZz48ZyBkYXRhLWNlbGwtaWQ9ImZldGNoIj48Zz48cmVjdCB4PSIzMjAiIHk9IjAiIHdpZHRoPSIxMjAiIGhlaWdodD0iNjAiIHJ4PSI5IiByeT0iOSIgZmlsbD0iI2ZmZjJjYyIgc3R5bGU9ImZpbGw6IGxpZ2h0LWRhcmsocmdiKDI1NSwgMjQyLCAyMDQpLCByZ2IoNDAsIDI5LCAwKSk7IHN0cm9rZTogbGlnaHQtZGFyayhyZ2IoMjE0LCAxODIsIDg2KSwgcmdiKDEwOSwgODEsIDApKTsiIHN0cm9rZT0iI2Q2YjY1NiIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjwvZz48Zz48ZyBmaWxsPSIjMDAwMDAwIiBzdHlsZT0iZmlsbDogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEycHgiPjx0ZXh0IHg9IjM4MCIgeT0iMzUiPkZldGNoIERhdGE8L3RleHQ+PC9nPjwvZz48L2c+PGcgZGF0YS1jZWxsLWlkPSJ2YWxpZGF0ZSI+PGc+PHJlY3QgeD0iNDYwIiB5PSIwIiB3aWR0aD0iMTgwIiBoZWlnaHQ9IjYwIiByeD0iOSIgcnk9IjkiIGZpbGw9IiNmOGNlY2MiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigyNDgsIDIwNiwgMjA0KSwgcmdiKDgxLCA0NSwgNDMpKTsgc3Ryb2tlOiBsaWdodC1kYXJrKHJnYigxODQsIDg0LCA4MCksIHJnYigyMTUsIDEyOSwgMTI2KSk7IiBzdHJva2U9IiNiODU0NTAiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48L2c+PGc+PGcgZmlsbD0iIzAwMDAwMCIgc3R5bGU9ImZpbGw6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyIgZm9udC1mYW1pbHk9IkhlbHZldGljYSIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMnB4Ij48dGV4dCB4PSI1NTAiIHk9IjM1Ij5WYWxpZGF0ZSAvIFByZXBhcmUgUmF3IERhdGE8L3RleHQ+PC9nPjwvZz48L2c+PGcgZGF0YS1jZWxsLWlkPSJwdWJsaXNoIj48Zz48cGF0aCBkPSJNIDcwMCAxNSBDIDY2OCAxNSA2NjAgMzAgNjg1LjYgMzMgQyA2NjAgMzkuNiA2ODguOCA1NCA3MDkuNiA0OCBDIDcyNCA2MCA3NzIgNjAgNzg4IDQ4IEMgODIwIDQ4IDgyMCAzNiA4MDAgMzAgQyA4MjAgMTggNzg4IDYgNzYwIDEyIEMgNzQwIDMgNzA4IDMgNzAwIDE1IFoiIGZpbGw9IiNlMWQ1ZTciIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigyMjUsIDIxMywgMjMxKSwgcmdiKDU3LCA0NywgNjMpKTsgc3Ryb2tlOiBsaWdodC1kYXJrKHJnYigxNTAsIDExNSwgMTY2KSwgcmdiKDE0OSwgMTE5LCAxNjMpKTsiIHN0cm9rZT0iIzk2NzNhNiIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PC9nPjxnPjxnIGZpbGw9IiMwMDAwMDAiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigwLCAwLCAwKSwgcmdiKDI1NSwgMjU1LCAyNTUpKTsiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2EiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTJweCI+PHRleHQgeD0iNzQwIiB5PSIyOCI+UHVibGlzaCB0byBNUTwvdGV4dD48dGV4dCB4PSI3NDAiIHk9IjQyIj4oUmFiYml0TVEpPC90ZXh0PjwvZz48L2c+PC9nPjxnIGRhdGEtY2VsbC1pZD0iYTEiPjxnPjxwYXRoIGQ9Ik0gMTIwIDMwIEwgMTI5Ljc2IDMwIiBmaWxsPSJub25lIiBzdHJva2U9IiMwMDAwMDAiIHN0eWxlPSJzdHJva2U6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIHBvaW50ZXItZXZlbnRzPSJzdHJva2UiLz48cGF0aCBkPSJNIDEzNy43NiAzMCBMIDEyOS43NiAzNCBMIDEyOS43NiAyNiBaIiBmaWxsPSIjMDAwMDAwIiBzdHlsZT0iZmlsbDogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IHN0cm9rZTogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IiBzdHJva2U9IiMwMDAwMDAiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PC9nPjwvZz48ZyBkYXRhLWNlbGwtaWQ9ImEyIj48Zz48cGF0aCBkPSJNIDMwMCAzMCBMIDMwOS43NiAzMCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjMDAwMDAwIiBzdHlsZT0ic3Ryb2tlOiBsaWdodC1kYXJrKHJnYigwLCAwLCAwKSwgcmdiKDI1NSwgMjU1LCAyNTUpKTsiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0ic3Ryb2tlIi8+PHBhdGggZD0iTSAzMTcuNzYgMzAgTCAzMDkuNzYgMzQgTCAzMDkuNzYgMjYgWiIgZmlsbD0iIzAwMDAwMCIgc3R5bGU9ImZpbGw6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyBzdHJva2U6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyIgc3Ryb2tlPSIjMDAwMDAwIiBzdHJva2Utd2lkdGg9IjIiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjwvZz48L2c+PGcgZGF0YS1jZWxsLWlkPSJhMyI+PGc+PHBhdGggZD0iTSA0NDAgMzAgTCA0NDkuNzYgMzAiIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMDAwMCIgc3R5bGU9InN0cm9rZTogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IiBzdHJva2Utd2lkdGg9IjIiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgcG9pbnRlci1ldmVudHM9InN0cm9rZSIvPjxwYXRoIGQ9Ik0gNDU3Ljc2IDMwIEwgNDQ5Ljc2IDM0IEwgNDQ5Ljc2IDI2IFoiIGZpbGw9IiMwMDAwMDAiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigwLCAwLCAwKSwgcmdiKDI1NSwgMjU1LCAyNTUpKTsgc3Ryb2tlOiBsaWdodC1kYXJrKHJnYigwLCAwLCAwKSwgcmdiKDI1NSwgMjU1LCAyNTUpKTsiIHN0cm9rZT0iIzAwMDAwMCIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48L2c+PC9nPjxnIGRhdGEtY2VsbC1pZD0iYTQiPjxnPjxwYXRoIGQ9Ik0gNjQwIDMwIEwgNjQ5Ljc2IDMwIiBmaWxsPSJub25lIiBzdHJva2U9IiMwMDAwMDAiIHN0eWxlPSJzdHJva2U6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIHBvaW50ZXItZXZlbnRzPSJzdHJva2UiLz48cGF0aCBkPSJNIDY1Ny43NiAzMCBMIDY0OS43NiAzNCBMIDY0OS43NiAyNiBaIiBmaWxsPSIjMDAwMDAwIiBzdHlsZT0iZmlsbDogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IHN0cm9rZTogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IiBzdHJva2U9IiMwMDAwMDAiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PC9nPjwvZz48L2c+PC9nPjwvZz48L3N2Zz4=) Typical **Push-Data-Collector** flow (the collector waits for data from the provider) ![push collector flow](data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPCEtLSBEbyBub3QgZWRpdCB0aGlzIGZpbGUgd2l0aCBlZGl0b3JzIG90aGVyIHRoYW4gZHJhdy5pbyAtLT4KPCFET0NUWVBFIHN2ZyBQVUJMSUMgIi0vL1czQy8vRFREIFNWRyAxLjEvL0VOIiAiaHR0cDovL3d3dy53My5vcmcvR3JhcGhpY3MvU1ZHLzEuMS9EVEQvc3ZnMTEuZHRkIj4KPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJiYWNrZ3JvdW5kOiAjZmZmZmZmOyBiYWNrZ3JvdW5kLWNvbG9yOiBsaWdodC1kYXJrKCNmZmZmZmYsIHZhcigtLWdlLWRhcmstY29sb3IsICMxMjEyMTIpKTsgY29sb3Itc2NoZW1lOiBsaWdodCBkYXJrOyIgeG1sbnM6eGxpbms9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGxpbmsiIHZlcnNpb249IjEuMSIgd2lkdGg9Ijc0MXB4IiBoZWlnaHQ9IjYxcHgiIHZpZXdCb3g9Ii0wLjUgLTAuNSA3NDEgNjEiIGNvbnRlbnQ9IiZsdDtteGZpbGUgaG9zdD0mcXVvdDthcHAuZGlhZ3JhbXMubmV0JnF1b3Q7IGFnZW50PSZxdW90O01vemlsbGEvNS4wIChYMTE7IExpbnV4IHg4Nl82NDsgcnY6MTM2LjApIEdlY2tvLzIwMTAwMTAxIEZpcmVmb3gvMTM2LjAmcXVvdDsgdmVyc2lvbj0mcXVvdDsyOC4wLjYmcXVvdDsgc2NhbGU9JnF1b3Q7MSZxdW90OyBib3JkZXI9JnF1b3Q7MCZxdW90OyZndDsmI3hBOyAgJmx0O2RpYWdyYW0gbmFtZT0mcXVvdDtQYWdlLTEmcXVvdDsgaWQ9JnF1b3Q7bE1jNmxOUExMWmR6ZU42ZGtmWmwmcXVvdDsmZ3Q7JiN4QTsgICAgJmx0O214R3JhcGhNb2RlbCBkeD0mcXVvdDsxNDI2JnF1b3Q7IGR5PSZxdW90Ozc5NyZxdW90OyBncmlkPSZxdW90OzEmcXVvdDsgZ3JpZFNpemU9JnF1b3Q7MTAmcXVvdDsgZ3VpZGVzPSZxdW90OzEmcXVvdDsgdG9vbHRpcHM9JnF1b3Q7MSZxdW90OyBjb25uZWN0PSZxdW90OzEmcXVvdDsgYXJyb3dzPSZxdW90OzEmcXVvdDsgZm9sZD0mcXVvdDsxJnF1b3Q7IHBhZ2U9JnF1b3Q7MSZxdW90OyBwYWdlU2NhbGU9JnF1b3Q7MSZxdW90OyBwYWdlV2lkdGg9JnF1b3Q7ODUwJnF1b3Q7IHBhZ2VIZWlnaHQ9JnF1b3Q7MTEwMCZxdW90OyBtYXRoPSZxdW90OzAmcXVvdDsgc2hhZG93PSZxdW90OzAmcXVvdDsmZ3Q7JiN4QTsgICAgICAmbHQ7cm9vdCZndDsmI3hBOyAgICAgICAgJmx0O214Q2VsbCBpZD0mcXVvdDswJnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDtteENlbGwgaWQ9JnF1b3Q7MSZxdW90OyBwYXJlbnQ9JnF1b3Q7MCZxdW90OyAvJmd0OyYjeEE7ICAgICAgICAmbHQ7bXhDZWxsIGlkPSZxdW90O3Byb3ZpZGVyJnF1b3Q7IHZhbHVlPSZxdW90O0V4dGVybmFsIFByb3ZpZGVyJmFtcDsjeGE7KFB1c2hlcyBEYXRhKSZxdW90OyBzdHlsZT0mcXVvdDtzaGFwZT1oZXhhZ29uO2ZpbGxDb2xvcj0jZGFlOGZjO3N0cm9rZUNvbG9yPSM2YzhlYmY7Zm9udFNpemU9MTI7JnF1b3Q7IHBhcmVudD0mcXVvdDsxJnF1b3Q7IHZlcnRleD0mcXVvdDsxJnF1b3Q7Jmd0OyYjeEE7ICAgICAgICAgICZsdDtteEdlb21ldHJ5IHg9JnF1b3Q7MjAmcXVvdDsgeT0mcXVvdDsxNDAmcXVvdDsgd2lkdGg9JnF1b3Q7MTYwJnF1b3Q7IGhlaWdodD0mcXVvdDs2MCZxdW90OyBhcz0mcXVvdDtnZW9tZXRyeSZxdW90OyAvJmd0OyYjeEE7ICAgICAgICAmbHQ7L214Q2VsbCZndDsmI3hBOyAgICAgICAgJmx0O214Q2VsbCBpZD0mcXVvdDtlbmRwb2ludCZxdW90OyB2YWx1ZT0mcXVvdDtDb2xsZWN0b3IgRW5kcG9pbnQmYW1wOyN4YTsoSFRUUCAvIFdlYmhvb2spJnF1b3Q7IHN0eWxlPSZxdW90O3NoYXBlPXJlY3RhbmdsZTtyb3VuZGVkPTE7ZmlsbENvbG9yPSNkNWU4ZDQ7c3Ryb2tlQ29sb3I9IzgyYjM2Njtmb250U2l6ZT0xMjsmcXVvdDsgcGFyZW50PSZxdW90OzEmcXVvdDsgdmVydGV4PSZxdW90OzEmcXVvdDsmZ3Q7JiN4QTsgICAgICAgICAgJmx0O214R2VvbWV0cnkgeD0mcXVvdDsyMDAmcXVvdDsgeT0mcXVvdDsxNDAmcXVvdDsgd2lkdGg9JnF1b3Q7MTgwJnF1b3Q7IGhlaWdodD0mcXVvdDs2MCZxdW90OyBhcz0mcXVvdDtnZW9tZXRyeSZxdW90OyAvJmd0OyYjeEE7ICAgICAgICAmbHQ7L214Q2VsbCZndDsmI3hBOyAgICAgICAgJmx0O214Q2VsbCBpZD0mcXVvdDt2YWxpZGF0ZSZxdW90OyB2YWx1ZT0mcXVvdDtWYWxpZGF0ZSAvIFByZXBhcmUgUmF3IERhdGEmcXVvdDsgc3R5bGU9JnF1b3Q7c2hhcGU9cmVjdGFuZ2xlO3JvdW5kZWQ9MTtmaWxsQ29sb3I9I2Y4Y2VjYztzdHJva2VDb2xvcj0jYjg1NDUwO2ZvbnRTaXplPTEyOyZxdW90OyBwYXJlbnQ9JnF1b3Q7MSZxdW90OyB2ZXJ0ZXg9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSB4PSZxdW90OzQwMCZxdW90OyB5PSZxdW90OzE0MCZxdW90OyB3aWR0aD0mcXVvdDsxODAmcXVvdDsgaGVpZ2h0PSZxdW90OzYwJnF1b3Q7IGFzPSZxdW90O2dlb21ldHJ5JnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDsvbXhDZWxsJmd0OyYjeEE7ICAgICAgICAmbHQ7bXhDZWxsIGlkPSZxdW90O3B1Ymxpc2gmcXVvdDsgdmFsdWU9JnF1b3Q7UHVibGlzaCB0byBNUSZhbXA7I3hhOyhSYWJiaXRNUSkmcXVvdDsgc3R5bGU9JnF1b3Q7c2hhcGU9Y2xvdWQ7ZmlsbENvbG9yPSNlMWQ1ZTc7c3Ryb2tlQ29sb3I9Izk2NzNhNjtmb250U2l6ZT0xMjsmcXVvdDsgcGFyZW50PSZxdW90OzEmcXVvdDsgdmVydGV4PSZxdW90OzEmcXVvdDsmZ3Q7JiN4QTsgICAgICAgICAgJmx0O214R2VvbWV0cnkgeD0mcXVvdDs2MDAmcXVvdDsgeT0mcXVvdDsxNDAmcXVvdDsgd2lkdGg9JnF1b3Q7MTYwJnF1b3Q7IGhlaWdodD0mcXVvdDs2MCZxdW90OyBhcz0mcXVvdDtnZW9tZXRyeSZxdW90OyAvJmd0OyYjeEE7ICAgICAgICAmbHQ7L214Q2VsbCZndDsmI3hBOyAgICAgICAgJmx0O214Q2VsbCBpZD0mcXVvdDthMSZxdW90OyBzdHlsZT0mcXVvdDtlbmRBcnJvdz1ibG9jaztzdHJva2VXaWR0aD0yOyZxdW90OyBwYXJlbnQ9JnF1b3Q7MSZxdW90OyBzb3VyY2U9JnF1b3Q7cHJvdmlkZXImcXVvdDsgdGFyZ2V0PSZxdW90O2VuZHBvaW50JnF1b3Q7IGVkZ2U9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSByZWxhdGl2ZT0mcXVvdDsxJnF1b3Q7IGFzPSZxdW90O2dlb21ldHJ5JnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDsvbXhDZWxsJmd0OyYjeEE7ICAgICAgICAmbHQ7bXhDZWxsIGlkPSZxdW90O2EyJnF1b3Q7IHN0eWxlPSZxdW90O2VuZEFycm93PWJsb2NrO3N0cm9rZVdpZHRoPTI7JnF1b3Q7IHBhcmVudD0mcXVvdDsxJnF1b3Q7IHNvdXJjZT0mcXVvdDtlbmRwb2ludCZxdW90OyB0YXJnZXQ9JnF1b3Q7dmFsaWRhdGUmcXVvdDsgZWRnZT0mcXVvdDsxJnF1b3Q7Jmd0OyYjeEE7ICAgICAgICAgICZsdDtteEdlb21ldHJ5IHJlbGF0aXZlPSZxdW90OzEmcXVvdDsgYXM9JnF1b3Q7Z2VvbWV0cnkmcXVvdDsgLyZndDsmI3hBOyAgICAgICAgJmx0Oy9teENlbGwmZ3Q7JiN4QTsgICAgICAgICZsdDtteENlbGwgaWQ9JnF1b3Q7YTMmcXVvdDsgc3R5bGU9JnF1b3Q7ZW5kQXJyb3c9YmxvY2s7c3Ryb2tlV2lkdGg9MjsmcXVvdDsgcGFyZW50PSZxdW90OzEmcXVvdDsgc291cmNlPSZxdW90O3ZhbGlkYXRlJnF1b3Q7IHRhcmdldD0mcXVvdDtwdWJsaXNoJnF1b3Q7IGVkZ2U9JnF1b3Q7MSZxdW90OyZndDsmI3hBOyAgICAgICAgICAmbHQ7bXhHZW9tZXRyeSByZWxhdGl2ZT0mcXVvdDsxJnF1b3Q7IGFzPSZxdW90O2dlb21ldHJ5JnF1b3Q7IC8mZ3Q7JiN4QTsgICAgICAgICZsdDsvbXhDZWxsJmd0OyYjeEE7ICAgICAgJmx0Oy9yb290Jmd0OyYjeEE7ICAgICZsdDsvbXhHcmFwaE1vZGVsJmd0OyYjeEE7ICAmbHQ7L2RpYWdyYW0mZ3Q7JiN4QTsmbHQ7L214ZmlsZSZndDsmI3hBOyI+PGRlZnMvPjxyZWN0IGZpbGw9IiNmZmZmZmYiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKCNmZmZmZmYsIHZhcigtLWdlLWRhcmstY29sb3IsICMxMjEyMTIpKTsiIHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHg9IjAiIHk9IjAiLz48Zz48ZyBkYXRhLWNlbGwtaWQ9IjAiPjxnIGRhdGEtY2VsbC1pZD0iMSI+PGcgZGF0YS1jZWxsLWlkPSJwcm92aWRlciI+PGc+PHBhdGggZD0iTSA0MCAwIEwgMTIwIDAgTCAxNjAgMzAgTCAxMjAgNjAgTCA0MCA2MCBMIDAgMzAgWiIgZmlsbD0iI2RhZThmYyIgc3R5bGU9ImZpbGw6IGxpZ2h0LWRhcmsocmdiKDIxOCwgMjMyLCAyNTIpLCByZ2IoMjksIDQxLCA1OSkpOyBzdHJva2U6IGxpZ2h0LWRhcmsocmdiKDEwOCwgMTQyLCAxOTEpLCByZ2IoOTIsIDEyMSwgMTYzKSk7IiBzdHJva2U9IiM2YzhlYmYiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjwvZz48Zz48ZyBmaWxsPSIjMDAwMDAwIiBzdHlsZT0iZmlsbDogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEycHgiPjx0ZXh0IHg9IjgwIiB5PSIyOCI+RXh0ZXJuYWwgUHJvdmlkZXI8L3RleHQ+PHRleHQgeD0iODAiIHk9IjQyIj4oUHVzaGVzIERhdGEpPC90ZXh0PjwvZz48L2c+PC9nPjxnIGRhdGEtY2VsbC1pZD0iZW5kcG9pbnQiPjxnPjxyZWN0IHg9IjE4MCIgeT0iMCIgd2lkdGg9IjE4MCIgaGVpZ2h0PSI2MCIgcng9IjkiIHJ5PSI5IiBmaWxsPSIjZDVlOGQ0IiBzdHlsZT0iZmlsbDogbGlnaHQtZGFyayhyZ2IoMjEzLCAyMzIsIDIxMiksIHJnYigzMSwgNDcsIDMwKSk7IHN0cm9rZTogbGlnaHQtZGFyayhyZ2IoMTMwLCAxNzksIDEwMiksIHJnYig2OCwgMTEwLCA0NCkpOyIgc3Ryb2tlPSIjODJiMzY2IiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PC9nPjxnPjxnIGZpbGw9IiMwMDAwMDAiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigwLCAwLCAwKSwgcmdiKDI1NSwgMjU1LCAyNTUpKTsiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2EiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTJweCI+PHRleHQgeD0iMjcwIiB5PSIyOCI+Q29sbGVjdG9yIEVuZHBvaW50PC90ZXh0Pjx0ZXh0IHg9IjI3MCIgeT0iNDIiPihIVFRQIC8gV2ViaG9vayk8L3RleHQ+PC9nPjwvZz48L2c+PGcgZGF0YS1jZWxsLWlkPSJ2YWxpZGF0ZSI+PGc+PHJlY3QgeD0iMzgwIiB5PSIwIiB3aWR0aD0iMTgwIiBoZWlnaHQ9IjYwIiByeD0iOSIgcnk9IjkiIGZpbGw9IiNmOGNlY2MiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigyNDgsIDIwNiwgMjA0KSwgcmdiKDgxLCA0NSwgNDMpKTsgc3Ryb2tlOiBsaWdodC1kYXJrKHJnYigxODQsIDg0LCA4MCksIHJnYigyMTUsIDEyOSwgMTI2KSk7IiBzdHJva2U9IiNiODU0NTAiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48L2c+PGc+PGcgZmlsbD0iIzAwMDAwMCIgc3R5bGU9ImZpbGw6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyIgZm9udC1mYW1pbHk9IkhlbHZldGljYSIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMnB4Ij48dGV4dCB4PSI0NzAiIHk9IjM1Ij5WYWxpZGF0ZSAvIFByZXBhcmUgUmF3IERhdGE8L3RleHQ+PC9nPjwvZz48L2c+PGcgZGF0YS1jZWxsLWlkPSJwdWJsaXNoIj48Zz48cGF0aCBkPSJNIDYyMCAxNSBDIDU4OCAxNSA1ODAgMzAgNjA1LjYgMzMgQyA1ODAgMzkuNiA2MDguOCA1NCA2MjkuNiA0OCBDIDY0NCA2MCA2OTIgNjAgNzA4IDQ4IEMgNzQwIDQ4IDc0MCAzNiA3MjAgMzAgQyA3NDAgMTggNzA4IDYgNjgwIDEyIEMgNjYwIDMgNjI4IDMgNjIwIDE1IFoiIGZpbGw9IiNlMWQ1ZTciIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigyMjUsIDIxMywgMjMxKSwgcmdiKDU3LCA0NywgNjMpKTsgc3Ryb2tlOiBsaWdodC1kYXJrKHJnYigxNTAsIDExNSwgMTY2KSwgcmdiKDE0OSwgMTE5LCAxNjMpKTsiIHN0cm9rZT0iIzk2NzNhNiIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PC9nPjxnPjxnIGZpbGw9IiMwMDAwMDAiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigwLCAwLCAwKSwgcmdiKDI1NSwgMjU1LCAyNTUpKTsiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2EiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTJweCI+PHRleHQgeD0iNjYwIiB5PSIyOCI+UHVibGlzaCB0byBNUTwvdGV4dD48dGV4dCB4PSI2NjAiIHk9IjQyIj4oUmFiYml0TVEpPC90ZXh0PjwvZz48L2c+PC9nPjxnIGRhdGEtY2VsbC1pZD0iYTEiPjxnPjxwYXRoIGQ9Ik0gMTYwIDMwIEwgMTY5Ljc2IDMwIiBmaWxsPSJub25lIiBzdHJva2U9IiMwMDAwMDAiIHN0eWxlPSJzdHJva2U6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIHBvaW50ZXItZXZlbnRzPSJzdHJva2UiLz48cGF0aCBkPSJNIDE3Ny43NiAzMCBMIDE2OS43NiAzNCBMIDE2OS43NiAyNiBaIiBmaWxsPSIjMDAwMDAwIiBzdHlsZT0iZmlsbDogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IHN0cm9rZTogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IiBzdHJva2U9IiMwMDAwMDAiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PC9nPjwvZz48ZyBkYXRhLWNlbGwtaWQ9ImEyIj48Zz48cGF0aCBkPSJNIDM2MCAzMCBMIDM2OS43NiAzMCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjMDAwMDAwIiBzdHlsZT0ic3Ryb2tlOiBsaWdodC1kYXJrKHJnYigwLCAwLCAwKSwgcmdiKDI1NSwgMjU1LCAyNTUpKTsiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0ic3Ryb2tlIi8+PHBhdGggZD0iTSAzNzcuNzYgMzAgTCAzNjkuNzYgMzQgTCAzNjkuNzYgMjYgWiIgZmlsbD0iIzAwMDAwMCIgc3R5bGU9ImZpbGw6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyBzdHJva2U6IGxpZ2h0LWRhcmsocmdiKDAsIDAsIDApLCByZ2IoMjU1LCAyNTUsIDI1NSkpOyIgc3Ryb2tlPSIjMDAwMDAwIiBzdHJva2Utd2lkdGg9IjIiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjwvZz48L2c+PGcgZGF0YS1jZWxsLWlkPSJhMyI+PGc+PHBhdGggZD0iTSA1NjAgMzAgTCA1NjkuNzYgMzAiIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMDAwMCIgc3R5bGU9InN0cm9rZTogbGlnaHQtZGFyayhyZ2IoMCwgMCwgMCksIHJnYigyNTUsIDI1NSwgMjU1KSk7IiBzdHJva2Utd2lkdGg9IjIiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgcG9pbnRlci1ldmVudHM9InN0cm9rZSIvPjxwYXRoIGQ9Ik0gNTc3Ljc2IDMwIEwgNTY5Ljc2IDM0IEwgNTY5Ljc2IDI2IFoiIGZpbGw9IiMwMDAwMDAiIHN0eWxlPSJmaWxsOiBsaWdodC1kYXJrKHJnYigwLCAwLCAwKSwgcmdiKDI1NSwgMjU1LCAyNTUpKTsgc3Ryb2tlOiBsaWdodC1kYXJrKHJnYigwLCAwLCAwKSwgcmdiKDI1NSwgMjU1LCAyNTUpKTsiIHN0cm9rZT0iIzAwMDAwMCIgc3Ryb2tlLXdpZHRoPSIyIiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48L2c+PC9nPjwvZz48L2c+PC9nPjwvc3ZnPg==) ## 2. Core Principles of Data Collector Development[​](#2-core-principles-of-data-collector-development "Direct link to 2. Core Principles of Data Collector Development") Adhering to these principles ensures your data collector is maintainable, scalable, and resilient: ### 2.1. Modularity and Single Responsibility[​](#21-modularity-and-single-responsibility "Direct link to 2.1. Modularity and Single Responsibility") Each data collector should focus on a single data source or a specific data acquisition method. Avoid combining multiple disparate data sources into one collector. This promotes: * **Clearer Logic**: Easier to understand, debug, and test. * **Independent Deployment**: Changes to one source don't affect others. * **Scalability**: You can scale collectors for different sources independently. ### 2.2. Configuration Management[​](#22-configuration-management "Direct link to 2.2. Configuration Management") Externalize all configurable parameters (API keys, endpoints, polling intervals, bucket names) using environment variables. This keeps your code clean, allows for easy deployment across different environments (development, staging, production), and avoids hardcoding sensitive information. ### 2.3. Error Handling and Resilience[​](#23-error-handling-and-resilience "Direct link to 2.3. Error Handling and Resilience") Data collection often involves interacting with external systems that can be unreliable. Your collector must: * **Handle Network Errors**: Implement retries with exponential backoff for transient network issues. * **Validate Data**: Ensure retrieved data is in the expected format before processing. * **Log Errors Effectively**: Provide sufficient context for debugging. * **Fail Early**: Let the application crash if something goes wrong, the test and production environment handles restarts. ### 2.4. Observability (Logging, Tracing, Metrics)[​](#24-observability-logging-tracing-metrics "Direct link to 2.4. Observability (Logging, Tracing, Metrics)") For production systems, it's crucial to understand what your collector is doing. * **Logging**: Use structured logging (`slog` in Go) to record events, errors, and progress. * **Tracing**: Integrate with a distributed tracing system (like OpenTelemetry, as used by `opendatahub-go-sdk`) to visualize the flow of requests and identify bottlenecks across microservices. * **Metrics**: Expose metrics (e.g., number of successful polls, failed polls, data volume, processing time) for monitoring and alerting. ### 2.5. Scheduling and Triggering[​](#25-scheduling-and-triggering "Direct link to 2.5. Scheduling and Triggering") Determine how your collector will be activated: * **Polling**: Regularly checking a source for new data (e.g., using cron jobs, `time.Ticker`). * **Event-Driven**: Reacting to events (e.g., a webhook notification, a message on a specific queue). The S3 poller uses a cron scheduler, which is a common pattern for polling. ## 3. Anatomy of a Go Data Collector: S3 Poller Example[​](#3-anatomy-of-a-go-data-collector-s3-poller-example "Direct link to 3. Anatomy of a Go Data Collector: S3 Poller Example") Let's dissect the provided S3 poller example to understand its components and how it embodies the principles above. ### 3.1. Project Structure[​](#31-project-structure "Direct link to 3.1. Project Structure") A typical Go project for a data collector might look like this: ``` . ├── src/ │ ├── main.go \# Main application logic │ └── go.mod \# Go modules file │ └── go.sum \# Go modules checksums │ └── go.work \# Go workspace file (if part of a monorepo) │ └── go.work.sum \# Go workspace checksums ├── .env \# Local environment variables for development ├── Dockerfile \# Containerization instructions (often in infrastructure/docker) ├── docker-compose.yml \# Local development setup (often in collector's root) └── helm/ \# Helm charts for Kubernetes deployment └── your-collector-name.yaml ``` ### 3.2. Environment Variables and Configuration Management[​](#32-environment-variables-and-configuration-management "Direct link to 3.2. Environment Variables and Configuration Management") By design we require all configuration to be done via `env variables`.
This simplifies the develop cycle and deployment allowing us to use **.env** files for testing and **Helm values.yaml** for deployment. ### 3.3. SDK Integration (`opendatahub-go-sdk`)[​](#33-sdk-integration-opendatahub-go-sdk "Direct link to 33-sdk-integration-opendatahub-go-sdk") The `opendatahub-go-sdk` is central to simplifying collector development. * Go Code (main.go) ``` // ... imports ... func main() { // 1. Initialize SDK components (messaging, logging, telemetry) ms.InitWithEnv(context.Background(), "", &env) slog.Info("Starting data collector...") // Ensure telemetry traces are flushed on application panic defer tel.FlushOnPanic() // ... AWS config and client setup ... // 2. Create a new data collector instance // dc.EmptyData indicates that the input channel doesn't carry specific data, // as the cron job just triggers a collection run. collector := dc.NewDc[dc.EmptyData](context.Background(), env.Env) // 3. Schedule data collection runs c := cron.New(cron.WithSeconds()) c.AddFunc(env.CRON, func() { // When the cron job triggers, send an empty input signal to the collector's channel. // This tells the collector to execute its data fetching logic. collector.GetInputChannel() <- dc.NewInput[dc.EmptyData](context.Background(), nil) }) slog.Info("Setup complete. Starting cron scheduler") go func() { c.Run() // Start the cron scheduler in a goroutine }() // 4. Start the collector's main processing loop // The provided function (anonymous func) is the core data collection logic. err = collector.Start(context.Background(), func(ctx context.Context, a dc.EmptyData) (*rdb.RawAny, error) { // This function is executed each time an input is received on the collector's channel. // It's responsible for fetching raw data and returning it in the SDK's RawAny format. // ... S3 data fetching logic ... var raw any if env.RAW_BINARY { raw = body // Store raw bytes } else { raw = string(body) // Store as string } // Return the collected raw data in the SDK's standardized format return &rdb.RawAny{ Provider: env.PROVIDER, // Unique identifier for the data source Timestamp: time.Now(), // Timestamp of data collection Rawdata: raw, // The actual raw data }, nil }) // 5. Handle fatal errors from the collector's main loop ms.FailOnError(context.Background(), err, err.Error()) } ``` **Key SDK Components:** * **`ms.InitWithEnv(ctx, "", &env)`**: This function is the SDK's entry point for initialization. It sets up: * **Structured Logging (`slog`)**: Configures logging based on `LOG_LEVEL`. * **Messaging**: Initializes the RabbitMQ client using `MQ_URI`, `MQ_CLIENT`, `MQ_EXCHANGE`. The SDK handles publishing raw data to the message queue. * **Telemetry**: Sets up OpenTelemetry for tracing and metrics, using `TELEMETRY_TRACE_GRPC_ENDPOINT` (seen in Helm config). * **`tel.FlushOnPanic()`**: Ensures that any buffered telemetry data (traces, metrics) is sent to the configured endpoint before the application crashes due to a panic. This is crucial for debugging production issues. * **`dc.NewDc[dc.EmptyData](ctx, env.Env)`**: Creates a new data collector instance. The `[dc.EmptyData]` type parameter indicates that the input to the collector's processing function doesn't carry specific data; it's simply a trigger. * **`collector.GetInputChannel()`**: Returns a channel that you can send signals to. Each signal (in this case, `dc.NewInput[dc.EmptyData](context.Background(), nil)`) triggers the `collector.Start` function's callback. This decouples the scheduling mechanism from the core collection logic. * **`collector.Start(ctx, func(ctx context.Context, a dc.EmptyData) (*rdb.RawAny, error))`**: This is the main processing loop of the collector. * It takes a `context.Context` and a callback function. * The callback function is where your actual data fetching logic resides. * It's expected to return an `*rdb.RawAny` object (the standardized raw data format) or an error. * The SDK then takes this `RawAny` object and publishes it to the configured message queue. * **`rdb.RawAny`**: This struct defines the standard format for raw data that the SDK expects. It includes `Provider` (a unique identifier for the data source), `Timestamp` (when the data was collected), and `Rawdata` (the actual fetched content). * **`ms.FailOnError(ctx, err, msg)`**: A utility function from the SDK that logs a fatal error and exits the application if `err` is not `nil`. This is used for unrecoverable errors during setup or the main loop. #### SDK Env Variables[​](#sdk-env-variables "Direct link to SDK Env Variables") When using the official SDK, there are some special env variables used to configure various aspects of the collector | Variable | Description | Allowed Values | Default | | ----------------------------------- | ---------------------------------------------------- | ----------------------------------------- | ---------------- | | `PROVIDER` | String identifying the data provider | \[path1]/\[path2]/... | (No default set) | | `MQ_URI` | Connection URI to connect to RabbitMQ | - | (No default set) | | `MQ_EXCHANGE` | Exchange name where to push messages | - | `ingress` | | `MQ_CLIENT` | Client name to identify the connection in RabbitMQ | - | (No default set) | | `LOG_LEVEL` | Sets the logging severity level. | `DEBUG`, `INFO`, `WARN`, `ERROR` | `INFO` | | `SERVICE_NAME` | The name of the service. | - | `gotel` | | `SERVICE_VERSION` | The version of the service. | - | `0.0.1` | | `TELEMETRY_ENABLED` | Enables or disables telemetry. | `true`, `false` | `true` | | `TELEMETRY_TRACE_ENABLED` | Enables or disables trace telemetry. | `true`, `false` | `true` | | `TELEMETRY_TRACE_BATCH_SIZE` | The maximum number of traces to be batched. | - | `10` | | `TELEMETRY_TRACE_BATCH_TIMEOUT_SEC` | The timeout for a trace batch in seconds. | - | `5` | | `TELEMETRY_TRACE_GRPC_ENDPOINT` | The gRPC endpoint for trace export. | - | `localhost:4317` | | `TELEMETRY_TRACE_TLS_ENABLED` | Enables or disables TLS for trace export. | `true`, `false` | `false` | | `TELEMETRY_TRACE_TLS_CERT` | Path to the TLS certificate file for trace export. | - | `""` | | `TELEMETRY_METRICS_ENABLED` | Enables or disables metrics telemetry. | `true`, `false` | `false` | | `TELEMETRY_METRICS_TIMEOUT_SEC` | The timeout for metrics export in seconds. | - | `30` | | `TELEMETRY_METRICS_INTERVAL_SEC` | The interval between metrics exports in seconds. | - | `60` | | `TELEMETRY_METRICS_GRPC_ENDPOINT` | The gRPC endpoint for metrics export. | - | `localhost:4317` | | `TELEMETRY_METRICS_TLS_ENABLED` | Enables or disables TLS for metrics export. | `true`, `false` | `false` | | `TELEMETRY_METRICS_TLS_CERT` | Path to the TLS certificate file for metrics export. | - | `""` | | `TELEMETRY_METRICS_EXPORT_MODE` | The export mode for metrics. | `otlp`, `prometheus` (implied by context) | `otlp` | | `TELEMETRY_METRICS_EXPORT_PORT` | The port for the metrics exporter. | - | `2112` | ### 3.4. Data Collection Logic (S3 Interaction)[​](#34-data-collection-logic-s3-interaction "Direct link to 3.4. Data Collection Logic (S3 Interaction)") The core logic of fetching data from S3 is encapsulated within the anonymous function passed to `collector.Start`. info Each collector implements its own logic, the below is only an example * Go Code (main.go) ``` // ... // Create a custom AWS configuration customConfig, err := config.LoadDefaultConfig(context.Background(), config.WithRegion(env.AWS_REGION), // Configure AWS region from environment config.WithCredentialsProvider( // Provide static credentials from environment credentials.NewStaticCredentialsProvider(env.AWS_ACCESS_KEY_ID, env.AWS_ACCESS_SECRET_KEY, ""), ), ) ms.FailOnError(context.Background(), err, "failed to create AWS config") // Fatal error if config fails // Create an S3 client using the custom configuration s3Client := s3.NewFromConfig(customConfig) // ... err = collector.Start(context.Background(), func(ctx context.Context, a dc.EmptyData) (*rdb.RawAny, error) { // Get the object from S3 output, err := s3Client.GetObject(context.Background(), &s3.GetObjectInput{ Bucket: aws.String(env.AWS_S3_BUCKET_NAME), // S3 bucket name from environment Key: aws.String(env.AWS_S3_FILE_NAME), // S3 file key (name) from environment }) if err != nil { // Log the error with context (bucket, file) and return nil, err to the SDK slog.Error("error while getting s3 object:", "err", err, "bucket", env.AWS_S3_BUCKET_NAME, "file", env.AWS_S3_FILE_NAME) return nil, err } defer output.Body.Close() // Ensure the S3 response body is closed body, err := io.ReadAll(output.Body) // Read the entire S3 object body if err != nil { slog.Error("error reading response body:", "err", err) return nil, err } var raw any // Declare a variable to hold the raw data if env.RAW_BINARY { raw = body // If RAW_BINARY is true, store the raw bytes } else { raw = string(body) // Otherwise, convert bytes to string } // Construct the standardized RawAny object return &rdb.RawAny{ Provider: env.PROVIDER, // Identifier for this data source Timestamp: time.Now(), // Timestamp of when the data was collected Rawdata: raw, // The actual raw data (string or []byte) }, nil }) ms.FailOnError(context.Background(), err, err.Error()) // Fatal error if collector.Start fails } ``` **Critical Points:** * **AWS SDK Initialization**: The AWS SDK is initialized with the `AWS_REGION`, `AWS_ACCESS_KEY_ID`, and `AWS_ACCESS_SECRET_KEY` from environment variables. This is a standard pattern for configuring cloud service clients. * **Error Handling**: Crucially, errors from `s3Client.GetObject` and `io.ReadAll` are logged with `slog.Error` (providing context like bucket and file names) and then returned. The `collector.Start` function will then handle these errors, potentially preventing the message from being acknowledged and allowing for retries or dead-lettering by the messaging system. * **`rdb.RawAny` Structure**: The returned `*rdb.RawAny` object is the standardized output of any data collector using this SDK. It's the payload that will be sent to the message queue. The `Provider` field is particularly important as it uniquely identifies the source of this raw data, allowing transformers to filter or process it accordingly. ## 4. Containerization with Docker[​](#4-containerization-with-docker "Direct link to 4. Containerization with Docker") Docker is the recommended way to package and deploy your data collector. The provided `Dockerfile` uses a multi-stage build to create efficient and secure images. * Dockerfile ``` # SPDX-FileCopyrightText: 2024 NOI Techpark # # SPDX-License-Identifier: CC0-1.0 FROM golang:1.25-bookworm AS base EXPOSE 8080 # Expose port 8080 (though this collector doesn't run a web server, it's a common practice) FROM base AS build-env WORKDIR /app COPY src/ . # Copy source code to the working directory RUN go mod download # Download Go module dependencies RUN CGO_ENABLED=0 GOOS=linux go build -o main # Build the Go application: # CGO_ENABLED=0: Disables CGo, producing a statically linked binary # GOOS=linux: Compiles for Linux operating system # -o main: Output executable named 'main' # BUILD published image (production-ready, minimal image) FROM alpine:3 AS build WORKDIR /app COPY --from=build-env /app/main . # Copy only the compiled binary from the build-env stage ENTRYPOINT [ "./main"] # Set the entrypoint for the container # LOCAL DEVELOPMENT (for hot-reloading/easier debugging) FROM base AS dev WORKDIR /code CMD ["go", "run", "."] # Run the Go application directly from source for development # TESTS FROM base AS test WORKDIR /code CMD ["go", "test", "."] # Run Go tests ``` warning Keep the `go` directive in your `go.mod` no higher than the Go version of this Dockerfile's base image (this guide uses `golang:1.25`). If you run `go mod init` / `go mod tidy` with a newer local Go, the directive is stamped to that version and the container build fails with `go.mod requires go >= 1.25.0 (running go 1.23.7)`. Either pin the `go` directive to match the image or bump the base image. **Critical Points:** * **Multi-Stage Build**: This is a best practice for Go applications. * `base`: Defines the base Go environment. * `build-env`: Used only for building the executable. It includes source code and dependencies. * `build`: The final, slim production image. It only copies the compiled binary from `build-env`, resulting in a much smaller image size (e.g., using `alpine:3` instead of the full `golang` image). This reduces attack surface and download times. * `dev` and `test`: Separate stages for local development and running tests, allowing for different build/run environments without polluting the production image. * **`CGO_ENABLED=0 GOOS=linux`**: This is crucial for Go applications in Docker. It ensures the Go binary is statically linked and compiled for Linux, making it highly portable and suitable for minimal base images like Alpine. ### 4.1. Local Orchestration with Docker Compose[​](#41-local-orchestration-with-docker-compose "Direct link to 4.1. Local Orchestration with Docker Compose") `docker-compose.yml` simplifies running and testing your collector and its dependencies (like RabbitMQ) locally. * docker-compose.yml ``` # SPDX-FileCopyrightText: 2024 NOI Techpark # # SPDX-License-Identifier: CC0-1.0 x-app-common: &app-common build: dockerfile: infrastructure/docker/Dockerfile context: . target: dev env_file: - .env volumes: - ./src:/code - ./infrastructure:/code/infrastructure - pkg:/go/pkg/mod working_dir: /code services: app-bdp: <<: *app-common profiles: - bdp networks: - ingestion app: <<: *app-common depends_on: rabbitmq: condition: service_healthy profiles: - dev rabbitmq: extends: file: ../lib/docker-compose/docker-compose.rabbitmq.yml service: rabbitmq attach: false profiles: - dev volumes: pkg: networks: ingestion: external: true ``` warning When developing you must careful to start the collector with `rabbitmq` service ``` docker compose --profile dev up ``` warning When testing the complete pipeline (comprehensive of the transformer) you need to start the full Open Data Hub Core and you must careful to start the collector without `rabbitmq` service ``` docker compose --profile bdp up ``` ## 5. Deployment with Helm[​](#5-deployment-with-helm "Direct link to 5. Deployment with Helm") Helm is a package manager for Kubernetes, used to define, install, and upgrade complex Kubernetes applications. The `helm/meteorology-bz-forecast.test.yaml` file is a values file that overrides default settings in a Helm chart for a specific deployment environment (e.g., a test environment). * Helm Chart Values (.test.yaml) ``` image: repository: ghcr.io/noi-techpark/opendatahub-collectors/dc-s3-poller # Docker image repository pullPolicy: IfNotPresent # Only pull image if not already present on the node tag: "0.0.1" # Specific image tag/version to deploy env: # Environment variables passed directly to the container MQ_CLIENT: dc-meteorology-bz-forecast PROVIDER: s3-poller/meteorology-bz-forecast CRON: "0 0 0/1 * * *" # Production cron schedule (hourly) AWS_REGION: "eu-west-1" AWS_S3_FILE_NAME: "SMOS_MCPL-WX_EXP_SIAG.JSON" AWS_S3_BUCKET_NAME: dc-meteorology-province-forecast SERVICE_NAME: dc-meteorology-bz-forecast # Service name for observability/discovery TELEMETRY_TRACE_GRPC_ENDPOINT: tempo-distributor-discovery.monitoring.svc.cluster.local:4317 # OpenTelemetry collector endpoint envSecretRef: # Reference Kubernetes secrets for sensitive environment variables - name: MQ_URI # Name of the environment variable secret: rabbitmq-svcbind # Name of the Kubernetes secret key: uri # Key within the secret that holds the value ``` **Critical Points:** * **`image`**: Defines which Docker image to deploy. * `repository`: The path to your Docker image (e.g., in GitHub Container Registry). * `pullPolicy`: `IfNotPresent` is common for test/dev, `Always` for production to ensure the latest image is used. * `tag`: Crucial for versioning and reproducibility. Always use a specific tag, never `latest` in production. * **`env`**: Directly sets environment variables within the Kubernetes pod. These values typically override defaults defined in the main Helm chart. Notice how `CRON` is set to an hourly schedule here, different from the local development `.env` file. * **`envSecretRef`**: **This is paramount for security in production environments.** Instead of hardcoding sensitive values like `MQ_URI` directly in the Helm values (which would be stored in Git), `envSecretRef` tells Kubernetes to fetch the value for `MQ_URI` from a Kubernetes Secret named `rabbitmq-svcbind` under the key `uri`. This ensures sensitive data is not exposed in configuration files. * **`SERVICE_NAME` and `TELEMETRY_TRACE_GRPC_ENDPOINT`**: These are used for integrating with the Kubernetes cluster's observability stack. `SERVICE_NAME` helps identify the service in monitoring tools, and `TELEMETRY_TRACE_GRPC_ENDPOINT` points to the OpenTelemetry collector or agent within the cluster, enabling distributed tracing. ## 6. Local Development Workflow[​](#6-local-development-workflow "Direct link to 6. Local Development Workflow") To get your S3 poller data collector running locally: 1. **Clone Repositories**: * `opendatahub-collectors` (contains your collector's source) * `infrastructure-v2` (contains the shared `docker-compose.rabbitmq.yml` and overall infrastructure compose files) 2. **Navigate to Collector Directory**: `cd opendatahub-collectors/collectors/s3-poller` (or your specific collector's path). 3. **Create `.env`**: Copy the provided `.env` content into a file named `.env` in this directory. Fill in your AWS credentials (`AWS_ACCESS_KEY_ID`, `AWS_ACCESS_SECRET_KEY`). 4. **Start Infrastructure**: From the `infrastructure-v2` directory, run the base and timeseries compose files: ``` cd ../../infrastructure-v2 # Adjust path if needed docker compose -f docker-compose.yml up -d # Run in detached mode docker compose -f docker-compose.timeseries.yml up -d ``` 5. **Start Collector**: From your collector's directory (`opendatahub-collectors/collectors/s3-poller`), run its `docker-compose.yml`: ``` cd opendatahub-collectors/collectors/s3-poller # Adjust path if needed docker compose --profile bdp up --build # Build and run your collector ``` ### Testing the Data Flow[​](#testing-the-data-flow "Direct link to Testing the Data Flow") * **Check Collector Logs**: Observe the logs of your `app` container. You should see messages like "Starting data collector...", "Polling S3 bucket...", and "Published dummy S3 data to message queue." (if using the dummy example). * **RabbitMQ Management**: Access `http://localhost:15673` (guest/guest) in your browser. Navigate to "Queues" and check the `ingress` queue (or whatever `MQ_EXCHANGE` you configured). You should see messages accumulating there. * **MongoDB**: Connect to `mongodb://localhost:27017/?directConnection=true` using a tool like MongoDB Compass. You can inspect the raw data stored by the SDK before it's picked up by a transformer. This setup allows you to develop and test your data collector in an environment that closely mirrors the production Kubernetes cluster, ensuring smooth integration with the Open Data Hub ecosystem. --- # Developing a Data Transformer from Scratch A data transformer is the bridge between the raw data a collector produces and the standardized entities the Open Data Hub serves. It consumes raw data events from the message queue, transforms them, and pushes the result to the Open Data Hub. **Where** it pushes depends on the kind of data: * **Time series measurements** go to the Timeseries Writer (BDP). See [Pushing to the Timeseries Writer](/data-ingestion/data-transformer-from-scratch/timeseries.md). * **Structured content** (announcements, points of interest, events, and similar) goes to the Content API. See [Pushing to the Content API](/data-ingestion/data-transformer-from-scratch/content.md). This page covers the parts **every** transformer shares: the role, the core principles, the SDK listener, configuration, containerization, and the local development workflow. The two pages above cover the write side specific to each target. ## 1. Understanding the Role of a Data Transformer[​](#1-understanding-the-role-of-a-data-transformer "Direct link to 1. Understanding the Role of a Data Transformer") A data transformer acts as a processing unit within the data integration pipeline. Its primary responsibilities include: * **Raw Data Consumption**: Listening to a message queue for events indicating new raw data. * **Raw Data Retrieval**: Fetching the actual raw data from the raw data storage (the SDK handles this when only an event notification was received). * **Data Transformation**: Converting the raw, source-specific data into the standardized data model of the Open Data Hub. This often involves: * **Parsing**: Deserializing the raw data (e.g., JSON, XML). * **Enrichment**: Adding derived information or linking to external datasets (e.g., geocoding, unit conversions). * **Mapping**: Translating source-specific identifiers or schemas to Open Data Hub standards. * **Publication**: Pushing the transformed data to its target (the Timeseries Writer for measurements, or the Content API for content). The transformer is where the "intelligence" of data standardization resides, making raw data consumable for various applications. ![data transformer flow](/assets/images/transformer-flow-558352fd2b18270b0f00ec654efe96aa.svg) ## 2. Core Principles of Data Transformer Development[​](#2-core-principles-of-data-transformer-development "Direct link to 2. Core Principles of Data Transformer Development") Many principles from data collector development apply, with some specific nuances for transformers: ### 2.1. Idempotency is Key[​](#21-idempotency-is-key "Direct link to 2.1. Idempotency is Key") Transformers *must* be idempotent. Processing the same raw data event multiple times should produce the same result without creating duplicates or conflicting states. Your transformation logic should avoid side effects from re-processing. ### 2.2. Robust Data Validation and Error Handling[​](#22-robust-data-validation-and-error-handling "Direct link to 2.2. Robust Data Validation and Error Handling") Data from collectors can be malformed or incomplete. Your transformer needs to: * **Validate Input**: Check if the raw data conforms to expected schemas. * **Selective Processing**: If a single record within a batch fails, decide whether to skip only that record or the entire batch. * **Dead-Letter Queues (DLQ)**: Failed messages should typically be moved to a DLQ for later inspection, preventing them from blocking the main queue. The SDK's consumer often handles this. * **Fail Early**: Let the application crash if something goes wrong; the test and production environment handles restarts. ### 2.3. Data Model Consistency[​](#23-data-model-consistency "Direct link to 2.3. Data Model Consistency") Strictly follow the Open Data Hub data model for your target: stations, data types, and measurements for time series; entities and tags for content. Assign stable, unique IDs and populate metadata richly (for example names in multiple languages) to improve discoverability. ### 2.4. Performance and Batching[​](#24-performance-and-batching "Direct link to 2.4. Performance and Batching") Transformations can be CPU or I/O intensive. Consider batch processing (both writer APIs support pushing data in batches), concurrency, and efficient in-memory lookups for static data instead of repeated file reads. ### 2.5. Observability[​](#25-observability "Direct link to 2.5. Observability") As with collectors, comprehensive logging, tracing, and metrics are crucial for monitoring the transformation process, identifying data quality issues, and debugging performance problems. ## 3. The transformer skeleton[​](#3-the-transformer-skeleton "Direct link to 3. The transformer skeleton") ### 3.1. Project structure[​](#31-project-structure "Direct link to 3.1. Project structure") A typical transformer project looks like this (static data files are added only when the transformation logic needs them): ``` . ├── docker-compose.yml # Local development setup ├── infrastructure │ ├── docker │ │ └── Dockerfile # Containerization instructions │ └── helm # Helm charts for Kubernetes deployment ├── resources # Static data files required by the transformer (optional) ├── src # Go source code │ ├── dto.go # Data Transfer Objects (raw data schema) │ ├── go.mod │ ├── main.go # Main application logic │ └── main_test.go # Unit tests └── testdata # Test input and expected output files ├── input └── output ``` ### 3.2. Configuration via environment variables[​](#32-configuration-via-environment-variables "Direct link to 3.2. Configuration via environment variables") By design, all configuration is done via **environment variables**. This simplifies the development cycle and deployment, letting you use a `.env` file for local testing and Helm `values.yaml` for deployment. ### 3.3. Consuming raw data with the SDK[​](#33-consuming-raw-data-with-the-sdk "Direct link to 3.3. Consuming raw data with the SDK") Every transformer uses the `opendatahub-go-sdk` to initialize common services and to consume raw data from the message queue. The skeleton is the same regardless of the write target: ``` func main() { ms.InitWithEnv(context.Background(), "", &env) // logging, messaging, telemetry defer tel.FlushOnPanic() // ... set up your write client (BDP or Content API) ... listener := tr.NewTr[YourDTO](context.Background(), env.Env) err := listener.Start(context.Background(), Transform) ms.FailOnError(context.Background(), err, "error while listening to queue") } func Transform(ctx context.Context, payload *rdb.Raw[YourDTO]) error { // map payload.Rawdata, then push to your target return nil } ``` * **`ms.InitWithEnv`** sets up logging, the messaging consumer, and telemetry based on `tr.Env`. * **`tr.NewTr[YourDTO]`** creates the listener; the type parameter is the Go struct the raw data is unmarshaled into (received as `rdb.Raw[YourDTO]`). * **`listener.Start(ctx, handler)`** consumes messages from the configured queue and calls your handler for each one. warning The type passed to `tr.NewTr[T]` is used to deserialize the raw data directly. If the collector stored the raw payload as a **serialized JSON string** (very common), construct the listener with `string` and wrap your handler with `RawString2JsonMiddleware`, which deserializes the string for you: ``` listener := tr.NewTr[string](context.Background(), env.Env) err := listener.Start(context.Background(), tr.RawString2JsonMiddleware[YourDTO](Transform)) ``` #### SDK environment variables[​](#sdk-environment-variables "Direct link to SDK environment variables") When using the official SDK, these environment variables configure the transformer: | Variable | Description | Allowed Values | Default | | --------------------------------- | ------------------------------------------------------------------- | -------------------------------- | ---------------- | | `PROVIDER` | String identifying the data provider | \[path1]/\[path2]/... | (No default set) | | `MQ_URI` | Connection URI to connect to RabbitMQ | - | (No default set) | | `MQ_CLIENT` | Client name to identify the connection in RabbitMQ | - | (No default set) | | `MQ_EXCHANGE` | The exchange notifying the new data | - | `routed` | | `MQ_QUEUE` | RabbitMQ queue this transformer pulls messages from | - | (No default set) | | `MQ_KEY` | Routing key used to route messages from `MQ_EXCHANGE` to `MQ_QUEUE` | path1.path2... | (No default set) | | `RAW_DATA_BRIDGE_ENDPOINT` | Endpoint of the Raw Data Bridge, which retrieves raw data | - | (No default set) | | `LOG_LEVEL` | Sets the logging severity level | `DEBUG`, `INFO`, `WARN`, `ERROR` | `INFO` | | `SERVICE_NAME` | The name of the service | - | `gotel` | | `SERVICE_VERSION` | The version of the service | - | `0.0.1` | | `TELEMETRY_ENABLED` | Enables or disables telemetry | `true`, `false` | `true` | | `TELEMETRY_TRACE_ENABLED` | Enables or disables trace telemetry | `true`, `false` | `true` | | `TELEMETRY_TRACE_GRPC_ENDPOINT` | The gRPC endpoint for trace export | - | `localhost:4317` | | `TELEMETRY_METRICS_ENABLED` | Enables or disables metrics telemetry | `true`, `false` | `false` | | `TELEMETRY_METRICS_GRPC_ENDPOINT` | The gRPC endpoint for metrics export | - | `localhost:4317` | The target-specific variables (the `BDP_*` settings for the Timeseries Writer, or the `ODH_CORE_*` settings for the Content API) are documented on the two target pages. ## 4. Containerization with Docker[​](#4-containerization-with-docker "Direct link to 4. Containerization with Docker") The Dockerfile uses a multi-stage build. Copy any static resource files into both the build and final stages. * Dockerfile ``` # SPDX-FileCopyrightText: 2024 NOI Techpark # # SPDX-License-Identifier: CC0-1.0 FROM golang:1.25-bookworm as base FROM base as build-env WORKDIR /app COPY src/. . # Copy source code COPY resources/. ./resources # Copy static resource files (if any) RUN go mod download RUN CGO_ENABLED=0 GOOS=linux go build -o main # BUILD published image (minimal, production-ready) FROM alpine:latest as build WORKDIR /app COPY --from=build-env /app/main . COPY --from=build-env /app/resources /resources ENTRYPOINT [ "./main"] # LOCAL DEVELOPMENT (for hot-reloading/easier debugging) FROM base as dev WORKDIR /code CMD ["go", "run", "main.go"] ``` warning Keep the `go` directive in your `go.mod` no higher than the Go version of this Dockerfile's base image (this guide uses `golang:1.25`), or the container build fails with `go.mod requires go >= ... (running go ...)`. ## 5. Local Orchestration with Docker Compose[​](#5-local-orchestration-with-docker-compose "Direct link to 5. Local Orchestration with Docker Compose") The transformer's `docker-compose.yml` needs a message queue (RabbitMQ) to consume from. The `dev` profile brings up its own RabbitMQ; the `bdp` profile runs against the shared `infrastructure-v2` stack on the external `ingestion` network. * docker-compose.yml ``` x-app-common: &app-common build: dockerfile: infrastructure/docker/Dockerfile context: . target: dev env_file: - .env volumes: - ./src:/code - ./infrastructure:/code/infrastructure - pkg:/go/pkg/mod working_dir: /code services: app-bdp: <<: *app-common profiles: - bdp networks: - ingestion app: <<: *app-common depends_on: rabbitmq: condition: service_healthy profiles: - dev rabbitmq: extends: file: ../lib/docker-compose/docker-compose.rabbitmq.yml service: rabbitmq attach: false profiles: - dev volumes: pkg: networks: ingestion: external: true ``` warning When developing standalone, start the transformer **with** its own `rabbitmq` service: ``` docker compose --profile dev up ``` warning When testing the complete pipeline (together with the collector and the full Open Data Hub Core), start the transformer **without** its own `rabbitmq` service: ``` docker compose --profile bdp up ``` ## 6. Local Development Workflow[​](#6-local-development-workflow "Direct link to 6. Local Development Workflow") 1. **Clone repositories**: `opendatahub-collectors` (your transformer's source) and [infrastructure-v2](https://github.com/noi-techpark/infrastructure-v2) (the shared compose files). 2. **Navigate** to your transformer's directory: `cd opendatahub-collectors/transformers/foo-bar`. 3. **Create `.env`**: copy the provided `.env` content and set the `MQ_*` queue settings, `RAW_DATA_BRIDGE_ENDPOINT`, and the target-specific credentials (see the [Timeseries](/data-ingestion/data-transformer-from-scratch/timeseries.md) or [Content](/data-ingestion/data-transformer-from-scratch/content.md) page). 4. **Start the infrastructure**: from the `infrastructure-v2` directory: ``` docker compose -f docker-compose.yml up -d docker compose -f docker-compose.timeseries.yml up -d ``` 5. **Start the transformer**: from your transformer's directory: ``` docker compose --profile bdp up --build ``` ### Testing the data flow[​](#testing-the-data-flow "Direct link to Testing the data flow") * **Transformer logs**: observe the logs of your `app` (or `app-bdp`) container. * **RabbitMQ Management**: `http://localhost:15673` (guest/guest). The `infrastructure-v2` stack maps the management UI to host port **15673**; a transformer or collector running with `--profile dev` would take `15672`. * **MongoDB**: `mongodb://localhost:27017/?directConnection=true`. Inspect the raw data stored by the SDK before it is picked up. ## 7. Choosing your write target[​](#7-choosing-your-write-target "Direct link to 7. Choosing your write target") The skeleton above is shared. The write side depends on the data: * **[Pushing to the Timeseries Writer (BDP)](/data-ingestion/data-transformer-from-scratch/timeseries.md)** for time series measurements from stations and sensors, using `go-bdp-client`. * **[Pushing to the Content API](/data-ingestion/data-transformer-from-scratch/content.md)** for structured content (announcements, points of interest, events), using the SDK's `clib` package. --- # Pushing to the Content API This page covers the write side of a transformer that produces **structured content** (announcements, points of interest, events, and similar), using the `clib` package of the `opendatahub-go-sdk`. For the shared transformer skeleton (the SDK listener, environment variables, containerization, and local run), see [Developing a Data Transformer from Scratch](/data-ingestion/data-transformer-from-scratch.md). This guide uses the `traffic-event-a22-opendata` transformer (which turns A22 motorway events into Content API `Announcement` entities) as the reference implementation. ## 1. The clib package[​](#1-the-clib-package "Direct link to 1. The clib package") `github.com/noi-techpark/opendatahub-go-sdk/clib` is the SDK's Content API client. Its `ContentAPI` interface exposes four operations: | Method | Purpose | | ------------------------------------------ | ----------------------------------------- | | `Get(ctx, apiPath, queryParams, out)` | Read entities (used by the cache loader). | | `Post(ctx, apiPath, queryParams, payload)` | Create an entity (used by tag sync). | | `Put(ctx, apiPath, id, payload)` | Update a single entity by its ID. | | `PutMultiple(ctx, apiPath, payload)` | Upsert a list of entities in one request. | warning `PutMultiple` (bulk upsert) is supported only by some entity endpoints. Where it is not available, write entities individually with `Put` (update by ID) or `Post` (create). The example in this guide uses `PutMultiple` because the `Announcement` endpoint supports it. Create the client with OAuth2 client credentials against the Core: ``` contentClient, err := clib.NewContentClient(clib.Config{ BaseURL: env.ODH_CORE_URL, // e.g. https://api.tourism.testingmachine.eu/v1 TokenURL: env.ODH_CORE_TOKEN_URL, // Keycloak token endpoint ClientID: env.ODH_CORE_TOKEN_CLIENT_ID, ClientSecret: env.ODH_CORE_TOKEN_CLIENT_SECRET, DisableOAuth: env.ODH_CORE_TOKEN_URL == "", // read-only access can skip auth }) ms.FailOnError(context.Background(), err, "failed to create content client") ``` `NewContentClient` fetches an initial token up front (so bad credentials fail fast), retries failed requests, and emits OpenTelemetry spans. The returned `ContentClient` is safe for concurrent use. info Reading from the Content API is open, so for a read-only run you can set `DisableOAuth: true` and omit the token settings. Creating or updating content requires client credentials authorized to write to the Content API; request them from the Open Data Hub team. ### Environment variables[​](#environment-variables "Direct link to Environment variables") In addition to the standard transformer variables (`MQ_*`, `LOG_LEVEL`, and telemetry, documented in [Developing a Data Transformer from Scratch](/data-ingestion/data-transformer-from-scratch.md)), a Content API transformer reads: | Variable | Description | | ------------------------------ | ------------------------------------------------------------------ | | `ODH_CORE_URL` | Content API base URL, including the `/v1` path. | | `ODH_CORE_TOKEN_URL` | OAuth2 token endpoint (Keycloak). Leave empty for a read-only run. | | `ODH_CORE_TOKEN_CLIENT_ID` | Client ID authorized to write content. | | `ODH_CORE_TOKEN_CLIENT_SECRET` | Client secret. | ## 2. Stateful memory with the cache (optional)[​](#2-stateful-memory-with-the-cache-optional "Direct link to 2. Stateful memory with the cache (optional)") Not every transformer needs this. If yours simply creates or overwrites entities, call `Post` or `Put` directly and skip to the next section. The cache is for transformers that must **detect changes** and keep a set of entities in sync: create new ones, update only the ones that changed, and end ones that disappeared from the source. The Content API itself is the state store, and `clib` rebuilds the cache from it on startup, so there is no separate database to manage. On startup, load the entities you already own into a cache with `LoadExisting`: ``` annCache, err = clib.LoadExisting(context.Background(), contentClient, clib.LoadConfig[Announcement]{ EntityType: "Announcement", QueryParams: map[string]string{ "active": "true", "source": "a22", "rawfilter": "isnotnull(Mapping.ProviderA22Open.Id)", }, IDFunc: func(a Announcement) string { return *a.ID }, }) ms.FailOnError(context.Background(), err, "failed to load announcements") ``` `LoadExisting` pages through the matching entities (using the Content API's `Items` / `TotalPages` envelope), hashes each one, and returns a `Cache[T]` keyed by your `IDFunc`. This is the transformer's memory of the current state, rebuilt from the Content API on every startup, so it survives restarts without a separate database. During transformation, use the cache to detect changes and push only what actually changed: ``` hash, changed, err := annCache.HasChanged(id, ann) if changed { annCache.Set(id, ann, hash) list = append(list, ann) // only new or changed entities } ``` `HasChanged` hashes the entity and compares it with the cached hash. Control which fields participate in the hash with struct tags: * `hash:"ignore"` on volatile or server-managed fields (timestamps, `_Meta`, IDs) so they do not trigger spurious updates. * `hash:"set"` on slices whose order is not significant (for example `TagIds`, `HasLanguage`). ``` type Generic struct { ID *string `json:"Id,omitempty" hash:"ignore"` Meta *clib.Metadata `json:"_Meta,omitempty" hash:"ignore"` Active bool `json:"Active"` TagIds []string `json:"TagIds,omitempty" hash:"set"` // ... } ``` warning Fields that participate in the hash must be concretely typed. A `map[string]any` does not hash consistently, so an entity that uses one would always look changed and be re-pushed on every run. In the example, the `Mapping` field is given an explicit struct type for exactly this reason. To detect entities that have **ended** (present in the cache but absent from the latest batch), iterate the cache after processing the batch: ``` for id, entry := range annCache.Entries() { if _, stillPresent := seen[id]; stillPresent { continue } ann := entry.Entity ann.EndTime = &sourceTime // mark it ended list = append(list, ann) annCache.Delete(id) } ``` ## 3. Deterministic IDs[​](#3-deterministic-ids "Direct link to 3. Deterministic IDs") Give every entity a stable ID derived from a unique source key, so that re-processing the same input updates the same entity instead of creating a duplicate: ``` const ID_TEMPLATE = "urn:announcements:a22" func generateOpendataID(event dto.A22OpendataEvent) string { return clib.GenerateID(ID_TEMPLATE, event.IDNotizia) // "urn:announcements:a22:{uuid5}" } ``` `GenerateID(prefix, input)` returns `{prefix}:{uuid5(input)}`. The UUID is a deterministic version 5 hash of the input, so the same input always yields the same ID. ## 4. Tag sync[​](#4-tag-sync "Direct link to 4. Tag sync") Tags are defined declaratively in a JSON file and synced to the Content API on startup. Each definition carries its multilingual names and the entity types it is valid for: resources/tags.json ``` [ { "id": "announcement:traffic-event", "name-it": "Evento di Traffico", "name-de": "Verkehrsereignis", "name-en": "Traffic Event", "types": ["announcement"] }, { "id": "traffic-event:road-work", "name-it": "Cantiere Stradale", "name-de": "Baustelle", "name-en": "Road Work", "types": ["announcement", "traffic-event"] } ] ``` ``` tags, err := clib.ReadTagDefs("../resources/tags.json") ms.FailOnError(context.Background(), err, "failed to read tags") err = clib.SyncTags(context.Background(), contentClient, tags, clib.SyncTagsConfig{ Source: "announcement", }) ms.FailOnError(context.Background(), err, "failed to sync tags") ``` `SyncTags` creates each tag with a `POST`, using the `id` from the definition as the tag's ID. Tags that already exist are ignored (the SDK returns `ErrAlreadyExists`, which `SyncTags` treats as success), so it is safe to run on every startup. Without an explicit `LicenseInfo`, tags default to `CC0`. ## 5. Putting it together[​](#5-putting-it-together "Direct link to 5. Putting it together") The `main` function wires the standard ingestion listener to the `clib` client, the cache, and tag sync: ``` func main() { ms.InitWithEnv(context.Background(), "", &env) slog.Info("Starting Content API transformer...") defer tel.FlushOnPanic() contentClient, _ = clib.NewContentClient(clib.Config{ /* ... */ }) annCache, _ = clib.LoadExisting(/* ... */) // stateful memory tags, _ = clib.ReadTagDefs("../resources/tags.json") clib.SyncTags(context.Background(), contentClient, tags, clib.SyncTagsConfig{Source: "announcement"}) listener := tr.NewTr[string](context.Background(), env.Env) listener.Start(context.Background(), tr.RawString2JsonMiddleware(Transform)) } ``` The body of `Transform` maps each raw event to an entity, skips unchanged entities via the cache, marks ended ones, and upserts the result: ``` func Transform(ctx context.Context, r *rdb.Raw[dto.Root]) error { // 1. map each raw event to an Announcement // 2. skip unchanged entities with annCache.HasChanged // 3. mark ended entities (in the cache but not in this batch) // 4. push the changed and ended entities if len(list) == 0 { return nil } return contentClient.PutMultiple(ctx, "Announcement", list) } ``` The listener is the same one every transformer uses; the right way to consume the raw data depends on **how the collector stored it** (`tr.NewTr[string]` with `RawString2JsonMiddleware` for a serialized JSON string, or `tr.NewTr[YourDTO]` directly for structured data). See [Developing a Data Transformer from Scratch](/data-ingestion/data-transformer-from-scratch.md#33-consuming-raw-data-with-the-sdk). Only the body of `Transform` and the `clib` write calls are specific to the Content API. ## 6. Testing[​](#6-testing "Direct link to 6. Testing") `clib` ships a mock of the `ContentAPI` interface, so you can test transformation logic without a live Core. Inject `clibmock.ContentMock` in place of the real client and assert on the calls it records: ``` import "github.com/noi-techpark/opendatahub-go-sdk/clib/clibmock" ``` This is the same input/output, golden-file approach used with `bdpmock` for time series transformers (the example ships `testdata/in*.json` and `out*.json`). ## 7. Reference implementation[​](#7-reference-implementation "Direct link to 7. Reference implementation") The complete, runnable example is `transformers/traffic-event-a22-opendata` in the [opendatahub-collectors](https://github.com/noi-techpark/opendatahub-collectors/tree/main/transformers/traffic-event-a22-opendata) monorepo. It shows mapping, hashing, ended-event detection, geo enrichment, the `clib` content model (built on `clib.Metadata`, `clib.LicenseInfo`, and `clib.GpsInfo`), and the full `.env`, Dockerfile, and Helm configuration. ## Related[​](#related "Direct link to Related") * [Developing a Data Transformer from Scratch](/data-ingestion/data-transformer-from-scratch.md) (the shared skeleton) * [Pushing to the Timeseries Writer (BDP)](/data-ingestion/data-transformer-from-scratch/timeseries.md) * [SDKs](/data-ingestion/sdks.md) * [Content API reference](/use-data/content-api/reference.md) --- # Pushing to the Timeseries Writer (BDP) This page covers the write side of a transformer that produces **time series measurements**, using the `go-bdp-client` SDK. For the shared transformer skeleton (the SDK listener, environment variables, containerization, and the local development workflow), see [Developing a Data Transformer from Scratch](/data-ingestion/data-transformer-from-scratch.md). ## 1. Authentication for the Timeseries Writer[​](#1-authentication-for-the-timeseries-writer "Direct link to 1. Authentication for the Timeseries Writer") The Timeseries Writer API uses OAuth2 for authentication. Transformers obtain an access token (via Keycloak, "Client Credentials" flow) to make authenticated requests. We provide shared OAuth client credentials for **development and testing purposes only**: * **Token Endpoint**: `https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token` * **Client ID**: `odh-mobility-datacollector-development` * **Client Secret**: `7bd46f8f-c296-416d-a13d-dc81e68d0830` The `odh-mobility-datacollector-development` client is authorized to write data to the Timeseries Writer. Important These credentials are strictly for **testing and development**. For production deployments, request dedicated OAuth client credentials from the Open Data Hub team. * Using the Go SDK * Self-managed When using the `go-bdp-client` SDK, authentication is largely transparent. You only configure the relevant environment variables: * `BDP_TOKEN_URL`: The URL of the OAuth2 token endpoint. * `BDP_CLIENT_ID`: The client ID for your transformer. * `BDP_CLIENT_SECRET`: The client secret for your transformer. info These `BDP_*` names apply to `go-bdp-client` v1.4 and later, which the boilerplate uses. Collectors or transformers pinned to `go-bdp-client` v1.3 or earlier read the equivalent `ODH_TOKEN_URL`, `ODH_CLIENT_ID`, and `ODH_CLIENT_SECRET` instead. The SDK (specifically `bdplib.FromEnv()`) automatically fetches an access token, refreshes it before it expires, and sets the `Authorization: Bearer` header on every request to the Timeseries Writer. For debugging, you can acquire a token manually: ``` curl -X POST \ "https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials&client_id=odh-mobility-datacollector-development&client_secret=7bd46f8f-c296-416d-a13d-dc81e68d0830" ``` Then include the returned `access_token` in the `Authorization` header, for example against `/json/stations`: ``` curl -X GET "http://localhost:8999/json/stations" \ --header 'Content-Type: application/json' \ --header 'Authorization: bearer YOUR_ACCESS_TOKEN' ``` ## 2. Initializing the BDP client[​](#2-initializing-the-bdp-client "Direct link to 2. Initializing the BDP client") A time series transformer initializes a `bdplib` client and synchronizes its data types on startup, then plugs the client into the [shared listener](/data-ingestion/data-transformer-from-scratch.md#33-consuming-raw-data-with-the-sdk): ``` func main() { ms.InitWithEnv(context.Background(), "", &env) slog.Info("Starting data transformer...") b := bdplib.FromEnv() // initialize the BDP client from BDP_* env vars defer tel.FlushOnPanic() // Register the data types this transformer will publish (once, on startup). dataTypeList := bdplib.NewDataTypeList(nil) err := dataTypeList.Load("datatypes.json") ms.FailOnError(context.Background(), err, "could not load datatypes") b.SyncDataTypes(dataTypeList.All()) // The BDP client is injected into the handler via a small wrapper. listener := tr.NewTr[Forecast](context.Background(), env) err = listener.Start(context.Background(), TransformWithBdp(b)) ms.FailOnError(context.Background(), err, "error while listening to queue") } // TransformWithBdp adapts the Transform logic to the SDK's tr.Handler signature, // injecting the BDP client. func TransformWithBdp(bdp bdplib.Bdp) tr.Handler[Forecast] { return func(ctx context.Context, payload *rdb.Raw[Forecast]) error { return Transform(ctx, bdp, payload) } } ``` * **`bdplib.FromEnv()`** reads the `BDP_` environment variables to configure the base URL, provenance details, and OAuth2 authentication. * **`b.SyncDataTypes(...)`** registers all data types this transformer can publish, so the target understands the metrics. It is typically called once on startup. * **`TransformWithBdp`** is an adapter: `tr.Handler` expects `func(ctx, *rdb.Raw[T]) error`, so the wrapper injects the `bdplib.Bdp` client into your `Transform`. ## 3. The Transform function[​](#3-the-transform-function "Direct link to 3. The Transform function") This is the core business logic: map the raw data into stations and measurements, then push them. info Each transformer implements its own logic; the example below (off-street parking) is illustrative. * Go Code (main.go) - Transform Function ``` const ( stationTypeParent = "ParkingFacility" // Parent station type stationType = "ParkingStation" // Child station type dataTypeFreeTotal = "free" dataTypeOccupiedTotal = "occupied" ) // Transform converts raw parking data into BDP stations and measurements. func Transform(ctx context.Context, bdp bdplib.Bdp, payload *rdb.Raw[FacilityData]) error { log := logger.Get(ctx) var parentStations []bdplib.Station stations := make(map[string]bdplib.Station) // child stations by ID dataMapParent := bdp.CreateDataMap() dataMap := bdp.CreateDataMap() ts := payload.Timestamp.UnixMilli() for _, facility := range payload.Rawdata { // 1. Parent station + metadata id := facility.GetID() parent_station_data := station_proto.GetStationByID(strconv.Itoa(id)) if parent_station_data == nil { log.Error("no parent station data", "facility_id", strconv.Itoa(id)) panic("no parent station data") } parentStation := bdplib.CreateStation( parent_station_data.ID, parent_station_data.Name, stationTypeParent, parent_station_data.Lat, parent_station_data.Lon, bdp.GetOrigin()) parentStation.MetaData = parent_station_data.ToMetadata() parentStations = append(parentStations, parentStation) // 2. Child stations + measurements for _, freePlace := range facility.FacilityDetails { facility_id := strconv.Itoa(facility.GetID()) + "_" + strconv.Itoa(freePlace.ParkNo) station_data := station_proto.GetStationByID(facility_id) if station_data == nil { log.Error("no station data", "facility_id", facility_id) panic("no station data") } station, ok := stations[facility_id] if !ok { station = bdplib.CreateStation( station_data.ID, station_data.Name, stationType, station_data.Lat, station_data.Lon, bdp.GetOrigin()) station.ParentStation = parentStation.Id // link child to parent station.MetaData = station_data.ToMetadata() stations[station_data.ID] = station } dataMap.AddRecord(station_data.ID, dataTypeFreeTotal, bdplib.CreateRecord(ts, freePlace.FreePlaces, 600)) dataMap.AddRecord(station_data.ID, dataTypeOccupiedTotal, bdplib.CreateRecord(ts, freePlace.CurrentLevel, 600)) } } // 3. Batch sync stations, then push measurements bdp.SyncStations(stationTypeParent, parentStations, true, true) bdp.SyncStations(stationType, values(stations), true, true) bdp.PushData(stationTypeParent, dataMapParent) bdp.PushData(stationType, dataMap) return nil } ``` **Critical points and patterns:** * **Hierarchical stations**: a parent station (`ParkingFacility`) groups child stations (`ParkingStation`); `station.ParentStation = parentStation.Id` establishes the relationship. * **`bdplib.CreateStation(id, name, type, lat, lon, origin)`** builds a station; assign `MetaData` to enrich it. * **`bdplib.CreateRecord(ts, value, period)`** creates a measurement record (`ts` is Unix milliseconds; `period` is the aggregation interval in seconds). * **`bdp.SyncStations(type, stations, syncState, onlyActivate)`**: `syncState=true` updates active/inactive state based on presence in the call; `onlyActivate=true` only activates existing stations (use `false` if new stations can appear dynamically). * **`bdp.PushData(type, dataMap)`** pushes all collected measurements for a station type in one batch call. The `dto.go` file defines the Go structs for the raw input schema (for example `FacilityData`) — this is the type passed to `tr.NewTr[FacilityData]`. ## 4. Deployment with Helm[​](#4-deployment-with-helm "Direct link to 4. Deployment with Helm") The Helm values configure the image and the environment, with the OAuth credentials loaded from Kubernetes secrets. * Helm Chart Values (Conceptual) ``` image: repository: ghcr.io/noi-techpark/opendatahub-transformers/tr-parking-offstreet-skidata pullPolicy: IfNotPresent tag: "0.0.1" env: LOG_LEVEL: "INFO" MQ_QUEUE: s3-poller.parking-offstreet-skidata MQ_EXCHANGE: routed MQ_KEY: s3-poller.parking-offstreet-skidata MQ_CLIENT: tr-parking-offstreet-skidata RAW_DATA_BRIDGE_ENDPOINT: "http://raw-data-bridge-service.default.svc.cluster.local:2000/" BDP_BASE_URL: https://share.opendatahub.testingmachine.eu BDP_PROVENANCE_VERSION: 0.1.0 BDP_PROVENANCE_NAME: tr-parking-offstreet-skidata BDP_ORIGIN: province-bolzano SERVICE_NAME: tr-parking-offstreet-skidata TELEMETRY_TRACE_GRPC_ENDPOINT: tempo-distributor-discovery.monitoring.svc.cluster.local:4317 envSecretRef: - name: MQ_URI secret: rabbitmq-svcbind key: uri - name: BDP_TOKEN_URL secret: odh-oauth-client-credentials key: token_url - name: BDP_CLIENT_ID secret: odh-oauth-client-credentials key: client_id - name: BDP_CLIENT_SECRET secret: odh-oauth-client-credentials key: client_secret ``` For production, `BDP_CLIENT_ID`, `BDP_CLIENT_SECRET`, and `BDP_TOKEN_URL` should always be loaded from Kubernetes secrets via `envSecretRef`, never committed to Git. ## 5. Testing with bdpmock[​](#5-testing-with-bdpmock "Direct link to 5. Testing with bdpmock") `go-bdp-client` ships `bdpmock`, which fully mocks the `bdplib.Bdp` interface so you can test the `Transform` function without a live writer. The pattern feeds a golden input file, runs `Transform` against the mock, and compares the recorded calls with an expected-output file: ``` func TestMyBestParking(t *testing.T) { var in = FacilityData{} station_proto = ReadStations("../resources/stations.csv") bdpmock.LoadInputData(&in, "../testdata/input/mybestparking.json") timestamp, _ := time.Parse("2006-01-02", "2025-01-01") raw := rdb.Raw[FacilityData]{Rawdata: in, Timestamp: timestamp} var out = bdpmock.BdpMockCalls{} bdpmock.LoadOutput(&out, "../testdata/output/mybestparking--out.json") b := bdpmock.MockFromEnv() // mock instead of bdplib.FromEnv() err := Transform(context.TODO(), b, &raw) require.Nil(t, err) req := b.(*bdpmock.BdpMock).Requests() NormalizeBdpMockCalls(&req) // sort slices for order-independent comparison testsuite.DeepEqualFromFile(t, out, req) } ``` * **`bdpmock.MockFromEnv()`** returns a mock `bdplib.Bdp` that records every `SyncStations`, `SyncDataTypes`, and `PushData` call instead of sending it. * **Input/output golden files** (`testdata/input/*.json`, `testdata/output/*.json`) make the test deterministic: the output file is the exact set of `bdplib` calls the transformer is expected to make. * **`NormalizeBdpMockCalls`** sorts the recorded slices so that `testsuite.DeepEqualFromFile` comparisons are order-independent. ## Related[​](#related "Direct link to Related") * [Developing a Data Transformer from Scratch](/data-ingestion/data-transformer-from-scratch.md) (the shared skeleton) * [Pushing to the Content API](/data-ingestion/data-transformer-from-scratch/content.md) * [SDKs](/data-ingestion/sdks.md) --- # Structure: A data integration pipeline consists of at least 2 microservices: * Data collector: collects raw data and puts it on a message queue * Transformer: * listens to a message queue for raw data events * transforms the raw data into API calls to opendatahub-timeseries-writer (BDP) So the flow will be `Data provider -> Data collector -> Message Queue -> Transformer -> Timeseries writer` Between data collector and transformer sits a raw data DB and API, the transformer only gets a notification event and has to request the actual raw data from the raw data API. This part is fully handled by the SDK. # Repos: * [Monorepo for data collectors and transformers](https://github.com/noi-techpark/opendatahub-collectors) * [Repo of target writer API that transformers write to](https://github.com/noi-techpark/opendatahub-timeseries-writer) * [Infrastructure repo with some docs and compose files](https://github.com/noi-techpark/infrastructure-v2) # Data collector: See . All new data collectors are currently developed in golang.
We provide a in-house SDK that simplifies development and standardizes concerns like telemetry, logging, and configuration. Attention: since the repo is evolving fast, some collectors are based on an outdated technology stack. Up to date are for example: * parking-offstreet-famas (proprietary XMLRPC API) * s3-poller (AWS S3) * rest-multi-poller (includes some more complicated telemetry stuff) When integrating a new data provider consider first if an already existing data collector can cover you use case or could with minor generic modifications. Currently there is no boilerplate generator for data collectors, you will have to start from scratch or by copying an existing one (we suggest s3-poller for simplicity). # Transformers See Attention: since the repo is evolving fast, some collectors are based on an outdated technology stack. Up to date are for example: * carsharing-alpsgo * parking-offstreet-skidata * meteorology-bz-forecast For transformers we provide a (somewhat untested) boilerplate generator.
You can find it at `/transformers/boilerplate/setup_go.sh` and it generates a minimal transformer service that you can extend.
The wizard will ask you for a two part provider tuple and Origin, you can use placeholders during development, they just have to match the tuple provided in the collectors. # Documentation writer API: Some documentation about writing legacy style data collectors can be found [here](https://github.com/noi-techpark/opendatahub-docs/wiki/Getting-started-with-a-new-Data-Collector-development).
The API has a [OpenAPI spec](https://swagger.opendatahub.com/?url=https://raw.githubusercontent.com/noi-techpark/bdp-core/main/openapi3.yml)
While the data collector architecture has changed, the principles of interacting with the writer API still remain the same. In golang, use the [go-bdp-client](https://github.com/noi-techpark/go-bdp-client) to interact with this API. Still, the documentation might be useful to understand the underlying data structures and calls. o # Local development ## Data collectors[​](#data-collectors "Direct link to Data collectors") Data collectors often already have a rabbitmq instance that starts up with them in their `docker-compose.yml` file. Access `http://localhost:15672` to check if messages are sent correctly ## The whole shebang[​](#the-whole-shebang "Direct link to The whole shebang") The [Infrastructure repo](https://github.com/noi-techpark/infrastructure-v2) provides docker compose files that replicate the Open Data Hub infrastructure for local development. There is a base compose `docker-compose.yml` that only starts the bare infrastructure without any domain specific APIs. ``` docker compose -f docker-compose.yml up ``` Note: You might see periodic error messages about `traces export: context deadline exceeded` due to telemetry systems not being active. Ignore these. For developing a Timeseries API integration you will in addition have to start the `docker-compose.timeseries.yml`, which includes inbound and outbound APIs your transformer interacts with. ``` docker compose -f docker-compose.timeseries.yml up ``` Now you should be able to point your collectors and transformers to these local endpoints. Make sure to run the collectors/transformers in `network_mode: host` if using docker compose. The boilerplate should already configure your transformer for this by default.
Your collector might need some config changes, because often when developing collectors, they have their own dockerized rabbitmq instance that conflicts with the infrastructure-v2 compose. ## Rabbitmq[​](#rabbitmq "Direct link to Rabbitmq") Rabbitmq runs on with credentials `guest/guest`.
Here you can check if queues are populated correctly of if some ## Mongodb[​](#mongodb "Direct link to Mongodb") Access the local mongodb with this url, you can use Mongodb Compass for instance mongodb://localhost:27017/?directConnection=true ## Authentication[​](#authentication "Direct link to Authentication") We provide a shared oauth client credentials `odh-mobility-writer-development` and `odh-mobility-datacollector-development` for server and client development respectively. The datacollector client is authorized to write to the writer client. Host:
Realm: noi
Token endpoint: client\_id: odh-mobility-datacollector-development
client\_secret: 7bd46f8f-c296-416d-a13d-dc81e68d0830 --- # Getting Started This guide provides a comprehensive overview of setting up your local development environment for Open Data Hub data collectors and transformers, focusing on Go-based implementations. You'll learn how to leverage Docker Compose for infrastructure, interact with essential endpoints, handle authentication for the Timeseries Writer, and get hands-on with creating and testing a collector and a transformer. ## 1. Understanding the Data Integration Pipeline[​](#1-understanding-the-data-integration-pipeline "Direct link to 1. Understanding the Data Integration Pipeline") A typical Open Data Hub data integration pipeline involves two primary microservices: * **Data Collector**: Responsible for collecting raw data from external data providers and publishing it to a message queue. * **Transformer**: Listens for raw data events from the message queue, transforms the raw data into the Open Data Hub format, and pushes it to the Open Data Hub: time series measurements to the `opendatahub-timeseries-writer` (BDP), or structured content to the `Content API`. The data flow is as follows: ![ingestion flow](/assets/images/ingestion-flow-bc5883d9af8b55bfd5be6174a3150734.svg) Between the data collector and transformer, a raw data database and API are used. The transformer receives a notification event and requests the actual raw data from the raw data API. This interaction is fully handled by the SDK. ## 2. Setting up your Local Development Environment[​](#2-setting-up-your-local-development-environment "Direct link to 2. Setting up your Local Development Environment") The [infrastructure-v2](https://github.com/noi-techpark/infrastructure-v2) repository provides Docker Compose files to replicate the Open Data Hub infrastructure locally. ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Docker * Docker Compose ### 2.1. Basic Infrastructure Setup[​](#21-basic-infrastructure-setup "Direct link to 2.1. Basic Infrastructure Setup") To start the core infrastructure without any domain-specific APIs: ``` git clone https://github.com/noi-techpark/infrastructure-v2.git cd infrastructure-v2 docker compose -f docker-compose.yml up ``` info You might see periodic error messages about `traces export: context deadline exceeded` due to telemetry systems not being active. These can be ignored. ### 2.2. Timeseries API Integration Setup[​](#22-timeseries-api-integration-setup "Direct link to 2.2. Timeseries API Integration Setup") For developing Timeseries API integrations (which your transformer will interact with), you also need to start `docker-compose.timeseries.yml`: ``` cd infrastructure-v2 # Ensure you are in the infrastructure-v2 repository docker compose -f docker-compose.timeseries.yml up ``` Now, your locally running collectors and transformers should be able to connect to these local endpoints. When running collectors/transformers using Docker Compose, ensure they are configured with `network_mode: host`. The boilerplate for transformers should handle this by default. Collectors might need some configuration adjustments if they typically bring up their own RabbitMQ instance, as this would conflict with the `infrastructure-v2`'s RabbitMQ. ### 2.3. Useful Local Endpoints[​](#23-useful-local-endpoints "Direct link to 2.3. Useful Local Endpoints") * **RabbitMQ Management Interface**: `http://localhost:15673` (credentials: `guest/guest`). Use this to monitor message queues. The `infrastructure-v2` stack maps the management UI to host port **15673**. A collector started with its own broker via `--profile dev` would take `15672`, so run collectors with `--profile bdp` against this shared broker to avoid a port conflict. * **MongoDB**: `mongodb://localhost:27017/?directConnection=true`. You can use a tool like MongoDB Compass to connect. ## 3. Getting Hands-on: Collector and Transformer Development (Golang)[​](#3-getting-hands-on-collector-and-transformer-development-golang "Direct link to 3. Getting Hands-on: Collector and Transformer Development (Golang)") All new data collectors and transformers are currently developed in Go. An [in-house SDK](https://github.com/noi-techpark/opendatahub-go-sdk) simplifies development by standardizing concerns like messaging, raw data retrival, telemetry, logging, and configuration. ### 3.1. Data Collectors[​](#31-data-collectors "Direct link to 3.1. Data Collectors") See the `collectors` directory in the monorepo: . There is currently no boilerplate generator for data collectors. You'll need to start from scratch or by copying an existing, up-to-date collector. For simplicity, `s3-poller` is a good starting point. info An in depth guide to develop a Data Collector from scratch is available [here](/data-ingestion/data-collector-from-scratch.md) info We also provide low-code Data Collector Blueprints! check out our [catalog](/data-ingestion/category/data-collector-blueprints) ### 3.2. Transformers[​](#32-transformers "Direct link to 3.2. Transformers") See the `transformers` directory in the monorepo: . For transformers, a boilerplate generator is available at `/transformers/boilerplate/setup_go.sh`. This script generates a minimal transformer service that you can extend. The wizard will ask for a two-part provider tuple and Origin, which need to match the tuple provided in the collectors. info An in depth guide to develop a Data Transformer from scratch is available [here](/data-ingestion/data-transformer-from-scratch.md) info Transformers can target either the Timeseries Writer (time series measurements) or the Content API (structured content such as announcements and points of interest). For the Content API pattern (the `clib` package, change-detection cache, and tag sync), see [Pushing to the Content API](/data-ingestion/data-transformer-from-scratch/content.md). info We also provide low-code Data Transformer Blueprints! check out our [catalog](/data-ingestion/category/data-transformer-blueprints) #### Example: Using the Transformer Boilerplate (Conceptual)[​](#example-using-the-transformer-boilerplate-conceptual "Direct link to Example: Using the Transformer Boilerplate (Conceptual)") ``` cd opendatahub-collectors/transformers/boilerplate ./setup_go.sh ``` Follow the prompts. For instance, if your collector is `my-data-collector` and the origin is `MyOrigin`, you might enter: * Provider Tuple: `my-data-collector` * Origin: `MyOrigin` This will generate a new directory (e.g., `transformers/my-data-transformer`) with a basic Go transformer structure. #### Key Aspects of a Go Transformer[​](#key-aspects-of-a-go-transformer "Direct link to Key Aspects of a Go Transformer") Transformers will: 1. **Listen to a Message Queue**: Consume raw data events. 2. **Fetch Raw Data**: Request the actual raw data from the raw data API (handled by SDK). 3. **Transform Data**: Convert the raw data into the format expected by the `opendatahub-timeseries-writer`. 4. **Push Data to Writer**: Use the `go-bdp-client` to send the transformed data. ### 3.3 Authentication for the Timeseries Writer (Transformers)[​](#33-authentication-for-the-timeseries-writer-transformers "Direct link to 3.3 Authentication for the Timeseries Writer (Transformers)") The Timeseries Writer API uses OAuth2 for authentication. Transformers need to obtain an access token to make authenticated requests. #### OAuth2 Client Credentials[​](#oauth2-client-credentials "Direct link to OAuth2 Client Credentials") We provide shared OAuth client credentials for development: * **Host**: `https://auth.opendatahub.testingmachine.eu/auth/` * **Realm**: `noi` * **Token Endpoint**: `https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token` * **Client ID**: `odh-mobility-datacollector-development` * **Client Secret**: `7bd46f8f-c296-416d-a13d-dc81e68d0830` The `odh-mobility-datacollector-development` client is authorized to write to the writer client. #### Using the Access Token in API Calls[​](#using-the-access-token-in-api-calls "Direct link to Using the Access Token in API Calls") Once you have the `access_token`, include it in the `Authorization` header of your HTTP requests to the Timeseries Writer API. ``` curl -X GET "http://localhost:8999/json/stations" \ --header 'Content-Type: application/json' \ --header 'Authorization: bearer YOUR_ACCESS_TOKEN' ``` --- # SDKs The Open Data Hub provides Software Development Kits (SDKs) to streamline the development of data integration components, such as data collectors and transformers. These SDKs abstract away common concerns like messaging, telemetry, and interaction with the Timeseries Writer and Content APIs, allowing developers to focus on the core business logic of data acquisition and transformation. Below is an overview of the available SDKs, categorized by their primary functional area. ## Ingestion[​](#ingestion "Direct link to Ingestion") The Ingestion SDKs provide frameworks and utilities for building microservices responsible for collecting raw data and transforming it into the Open Data Hub's standardized format. They offer abstractions for the lifecycle of data collectors and transformers, handling message consumption, raw data processing, and structured data output. | Language | Git Repository | Official Support | | -------- | ---------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | Go | [`github.com/noi-techpark/opendatahub-go-sdk/ingest/dc`](https://github.com/noi-techpark/opendatahub-go-sdk/tree/main/ingest/dc) | Yes | | Go | [`github.com/noi-techpark/opendatahub-go-sdk/ingest/tr`](https://github.com/noi-techpark/opendatahub-go-sdk/tree/main/ingest/tr) | Yes | | Go | [`github.com/noi-techpark/opendatahub-go-sdk/ingest/rdb`](https://github.com/noi-techpark/opendatahub-go-sdk/tree/main/ingest/rdb) | Yes | ## Messaging[​](#messaging "Direct link to Messaging") The Messaging SDK facilitates interaction with the underlying message queue system (e.g., RabbitMQ). It provides standardized interfaces for publishing raw data from collectors and consuming raw data events in transformers, ensuring reliable and asynchronous data flow within the pipeline. | Language | Git Repository | Official Support | | -------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | Go | [`github.com/noi-techpark/opendatahub-go-sdk/ingest/ms`](https://github.com/noi-techpark/opendatahub-go-sdk/tree/main/ingest/ms) | Yes | ## Telemetry[​](#telemetry "Direct link to Telemetry") The Telemetry SDK integrates observability features into your data integration components. It provides tools for structured logging, distributed tracing (compatible with OpenTelemetry), and potentially metrics, enabling better monitoring, debugging, and performance analysis of your microservices. | Language | Git Repository | Official Support | | -------- | ---------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | Go | [`github.com/noi-techpark/opendatahub-go-sdk/tel`](https://github.com/noi-techpark/opendatahub-go-sdk/tree/main/tel) | Yes | | Go | [`github.com/noi-techpark/opendatahub-go-sdk/tel/logger`](https://github.com/noi-techpark/opendatahub-go-sdk/tree/main/tel/logger) | Yes | ## Timeseries Writer API[​](#timeseries-writer-api "Direct link to Timeseries Writer API") The Timeseries Writer API SDK (also known as the BDP client) provides a convenient and type-safe way to interact with the `opendatahub-timeseries-writer` service. It simplifies operations such as authenticating, synchronizing data types and stations, and pushing time-series measurements in the Open Data Hub's standardized format. | Language | Git Repository | Official Support | | -------- | ---------------------------------------------------------------------------------------- | ---------------- | | Go | [`github.com/noi-techpark/go-bdp-client`](https://github.com/noi-techpark/go-bdp-client) | Yes | ## Content API[​](#content-api "Direct link to Content API") The Content API SDK (the `clib` package) is a client for the Open Data Hub Content API. It handles OAuth2 authentication against the Core, retries, and tracing, and provides helpers for transformers that push structured content: a generic entity client (`Get`, `Post`, `Put`, `PutMultiple`), an optional change-detection cache (`LoadExisting`, `Cache`), declarative tag sync (`ReadTagDefs`, `SyncTags`), and deterministic IDs (`GenerateID`). See [Pushing to the Content API](/data-ingestion/data-transformer-from-scratch/content.md). | Language | Git Repository | Official Support | | -------- | ---------------------------------------------------------------------------------------------------------------------- | ---------------- | | Go | [`github.com/noi-techpark/opendatahub-go-sdk/clib`](https://github.com/noi-techpark/opendatahub-go-sdk/tree/main/clib) | Yes | ## Testing[​](#testing "Direct link to Testing") The Testing SDKs and utilities are designed to simplify the creation of robust unit and integration tests for your data integration components. They often provide mock implementations of external services, allowing for deterministic and isolated testing of transformation logic and API interactions. | Language | Git Repository | Official Support | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | Go | [`github.com/noi-techpark/go-bdp-client/bdpmock`](https://github.com/noi-techpark/go-bdp-client/tree/main/bdpmock) | Yes | | Go | [`github.com/noi-techpark/opendatahub-go-sdk/testsuite`](https://github.com/noi-techpark/opendatahub-go-sdk/tree/main/testsuite) | Yes | | Go | [`github.com/noi-techpark/opendatahub-go-sdk/clib/clibmock`](https://github.com/noi-techpark/opendatahub-go-sdk/tree/main/clib/clibmock) | Yes | --- # Transformer Boilerplate Generator As of now, the Open Data Hub does not offer a wide collection of pre-configured "Transformer Blueprints" in the same way as some Data Collectors. Instead, we provide a **boilerplate generator** that sets up a minimal, ready-to-extend Go-based transformer service. This allows you to quickly get started with a functional transformer and then customize its core transformation logic to fit your specific data requirements. This boilerplate generator is a command-line script that automates the creation of the necessary project structure, Go module setup, and initial configuration files for a new transformer. ## Location of the Boilerplate Generator[​](#location-of-the-boilerplate-generator "Direct link to Location of the Boilerplate Generator") The boilerplate generator script is located within the `opendatahub-collectors` monorepo at: [`https://github.com/noi-techpark/opendatahub-collectors/tree/main/transformers/boilerplate`](https://github.com/noi-techpark/opendatahub-collectors/tree/main/transformers/boilerplate) ## How to Use the Boilerplate Generator[​](#how-to-use-the-boilerplate-generator "Direct link to How to Use the Boilerplate Generator") To create a new transformer using the boilerplate, follow these steps: 1. **Clone the `opendatahub-collectors` Repository**: If you haven't already, clone the monorepo to your local machine: ``` git clone https://github.com/noi-techpark/opendatahub-collectors.git cd opendatahub-collectors ``` 2. **Navigate to the Boilerplate Directory**: Change your current directory to where the `setup_go.sh` script is located: ``` cd transformers/boilerplate ``` 3. **Run the Setup Script**: Execute the `setup_go.sh` script. The script will prompt you for several pieces of information to customize your new transformer. ``` ./setup_go.sh ``` You will be asked for the following: * **Project name**: This will be the name of your transformer's root folder, used in CI/CD pipelines, and as part of its Kubernetes service name. * *Example*: `parking-valgardena` * **First part of provider tuple**: This is the first segment of the `PROVIDER` environment variable that your transformer will use. It typically identifies the data source category or the collector. * *Example*: `parking-offstreet` * **Second part of provider tuple**: This is the second segment of the `PROVIDER` environment variable, often identifying the specific data source or sub-category. * *Example*: `skidata` * **Origin**: This identifies the original source or lineage of the data. It's used in the BDP provenance. * *Example*: `valgardena` After providing the details, the script will ask for confirmation. Type `y` (or `Y`) and press Enter to proceed. **Example Interaction**: ``` This wizard will set up the boilerplate for a new golang transformer Project name. This will determine root folder, cicd and k8s service e.g. 'parking-valgardena': my-new-transformer First part of provider tuple: my-data-source Second part of provider tuple: my-dataset Origin: my-organization Are you sure these are correct (y/n)? y ok, proceeding... All setup! ``` ## Next Steps After Generation[​](#next-steps-after-generation "Direct link to Next Steps After Generation") After the boilerplate is set up, you will have a functional, albeit minimal, transformer. Your primary task will be to: 1. **Implement Transformation Logic**: Modify the `src/main.go` and potentially `src/dto.go` files in your new transformer's directory to implement the specific data parsing, validation, and mapping logic required to convert your raw data into the Open Data Hub's BDP format. 2. **Configure Environment Variables**: Copy `.env.example` to `.env` and fill in the values for your local development. If the generated `.env.example` uses `ODH_TOKEN_URL` / `ODH_CLIENT_ID` / `ODH_CLIENT_SECRET`, rename them to `BDP_TOKEN_URL` / `BDP_CLIENT_ID` / `BDP_CLIENT_SECRET`, which is what the bundled `go-bdp-client` (v1.4 and later) actually reads. For deployment, ensure your Helm values (e.g., in `infrastructure/helm/.yaml`) are correctly configured, especially for sensitive data using Kubernetes secrets. 3. **Test Your Transformer**: The generator does not scaffold a test file, so create a `main_test.go` using `bdpmock` (provided by `go-bdp-client`) to ensure your transformation logic is correct and robust. 4. **Deploy**: Use the generated Dockerfile and Helm charts to build and deploy your transformer to your development, testing, and production environments. This boilerplate significantly reduces the initial setup time, allowing you to focus immediately on the unique aspects of your data transformation. --- # Analytics [Open Data Hub Analytics](https://analytics.opendatahub.com) is a web tool for **visualizing Open Data Hub time series data** without writing code. It is useful for quickly seeing how a measurement behaves over time, comparing stations, and sharing a view with others. Use it to inspect data visually before or alongside building requests against the [Time Series API](/use-data/time-series-api/reference.md). ## 1. What you can do[​](#1-what-you-can-do "Direct link to 1. What you can do") * **Build charts** from mobility datasets by selecting a data provider, a category, and the values to plot. * **Choose a time range**: presets (today, yesterday and today, one week, one month) or a custom range. * **Filter** by data provider and category to focus on the stations you care about. * **Adjust the view**: change the plot height and toggle automatic refresh for live data. * **Share a configuration**: generate a permanent link to a specific chart for colleagues. ## 2. How it fits your workflow[​](#2-how-it-fits-your-workflow "Direct link to 2. How it fits your workflow") Analytics reads the same mobility data exposed by the Time Series API. A common pattern is to explore a measurement visually here, then reproduce it programmatically: * Read the data in your application through the [Time Series API](/use-data/time-series-api/reference.md). * For analysis in R, the [`bzar` R package](/use-data/authentication-and-access/quotas-and-tools.md) provides programmatic access to the same data. ## 3. Related[​](#3-related "Direct link to 3. Related") * [Discovery](/tools/discovery.md) and [Data Browser](/tools/data-browser/.md) * [Time Series API reference](/use-data/time-series-api/reference.md) * [Quotas, CLI and the R package](/use-data/authentication-and-access/quotas-and-tools.md) --- # Data Browser The Open Data Hub [Data Browser](https://databrowser.opendatahub.com/) is a web-based tool for exploring, filtering, and managing Open Data Hub datasets through a user-friendly interface. For API users it is a convenient way to see the exact data model and fields the Content API returns; for content editors it is also where tourism content is created and maintained. ## 1. Exploring and filtering data[​](#1-exploring-and-filtering-data "Direct link to 1. Exploring and filtering data") The Data Browser supports targeted exploration through filtering in **table view** (filtering is not available in other views). When multiple filters are applied, they are combined with AND logic, so all conditions must be satisfied for a record to appear. The results align with how the **Content API** returns filtered responses. For example, a filter on a `title` field corresponds to the `search` parameter in the Content API: * Go * curl * Python * JavaScript ``` // Search for events with "conference" in the title resp, err := http.Get("https://tourism.api.opendatahub.com/v1/Event?search=conference") ``` ``` curl "https://tourism.api.opendatahub.com/v1/Event?search=conference" ``` ``` import requests response = requests.get("https://tourism.api.opendatahub.com/v1/Event", params={"search": "conference"}) print(response.json()) ``` ``` fetch('https://tourism.api.opendatahub.com/v1/Event?search=conference') .then(response => response.json()) .then(data => console.log(data)); ``` info Filtering in the Data Browser reflects the same constraints as the Content API, where parameters like `search`, `tagfilter`, and field-specific filters refine results. See [Filtering and sorting](/use-data/content-api/filtering-and-sorting.md). ## 2. Managing content[​](#2-managing-content "Direct link to 2. Managing content") The Data Browser is also where editors create and maintain tourism content: events, images, tags, and news articles. Those editorial workflows are documented separately, since they are not needed to consume the APIs. See [Managing content in the Data Browser](/tools/data-browser/content-management.md). ## 3. Related[​](#3-related "Direct link to 3. Related") * [Content API reference](/use-data/content-api/reference.md) * [Filtering and sorting](/use-data/content-api/filtering-and-sorting.md) * [Discovery](/tools/discovery.md) and [Analytics](/tools/analytics.md) --- # Managing content in the Data Browser This guide covers the editorial workflows in the [Data Browser](https://databrowser.opendatahub.com/): maintaining events, images, tags, and news articles. It is aimed at content editors and authorized users. If you only need to read data through the APIs, see the [Data Browser overview](/tools/data-browser/.md) instead. ## 1. Managing event data[​](#1-managing-event-data "Direct link to 1. Managing event data") Events are managed through structured forms in the Data Browser, but the underlying data model informs how event data should be structured for consumption via the **Content API**. ### 1.1. Event duration and scheduling[​](#11-event-duration-and-scheduling "Direct link to 1.1. Event duration and scheduling") For accurate representation across channels, event timing must be carefully defined: * An event ending **before midnight** should have its end time set to `23:59` on the same date. * An event ending **at midnight** should have its end time set to `00:00` on the next calendar day. Setting an end time of `00:00` on the same day is incorrect, as it refers to the start of that day and may cause display issues or application errors. | Example | End Date | End Time | Notes | | ------------------------------- | ---------- | -------- | ----------------------------------------- | | July 11, 2024, ends at 23:59 | 2024-07-11 | 23:59 | Correct for pre-midnight end | | July 11, 2024, ends at midnight | 2024-07-12 | 00:00 | Correct for exact midnight end | | July 11, 2024, ends at 00:00 | 2024-07-11 | 00:00 | Incorrect: refers to the start of the day | warning Avoid setting event end times to `00:00` on the same day. Use `23:59` for same-day ends or `00:00` on the next day for true midnight boundaries. ### 1.2. Multi-day events[​](#12-multi-day-events "Direct link to 1.2. Multi-day events") Multi-day events must be represented as separate entries for each day, with individual room bookings created per day. This ensures correct scheduling visibility, especially on platforms like `today.noi.bz.it`, which rely on room-level data. Additionally: * Setup and dismantling periods require their own room bookings if they occur outside main event hours. * The **Content API** returns event duration based on the `DateBegin` and `DateEnd` fields, while room assignments are handled under `EventDetails.VenueRoom`. ## 2. Image handling in content records[​](#2-image-handling-in-content-records "Direct link to 2. Image handling in content records") Images associated with content such as events or articles follow specific formatting rules enforced by the Data Browser interface. These constraints ensure compatibility with downstream consumers of the **Content API**. ### 2.1. Image requirements[​](#21-image-requirements "Direct link to 2.1. Image requirements") | Property | Value | | ---------------------- | ------------------------------ | | Accepted formats | `jpg`, `png` | | Maximum file size | 4 MB | | Recommended dimensions | 1170x780 pixels | | Orientation | Landscape preferred | | First image | Used as the main display image | Images are uploaded through the Data Browser form interface using a "Browse" button, which triggers the browser's native file selector. Once uploaded, images can be reordered by drag-and-drop in the **Images** section. info The **Content API** returns image URLs in the `Images` array. The first entry is typically used as the primary image in client applications. ## 3. Tag management[​](#3-tag-management "Direct link to 3. Tag management") Tags are used to classify content across datasets in the **Content API**. They support multilingual labels and are referenced in fields like `Tags` and `tagfilter`. ### 3.1. Creating new tags[​](#31-creating-new-tags "Direct link to 3.1. Creating new tags") New tags can be added in the Data Browser by authorized users. Each tag must include: * Names in German (`de`), Italian (`it`), and English (`en`) * One or more types (e.g., `eventtopic`, `gastronomycategory`) * A list of entities (from `_Meta.Type`) for which the tag is valid * `Active on Source` set to `true` * Source attribution (e.g., `NOI AG`) To retrieve valid tag types: ``` GET https://tourism.api.opendatahub.com/v1/Distinct?type=tag&fields=Types.[*]&getasarray=true ``` To retrieve valid entity types: ``` GET https://tourism.api.opendatahub.com/v1/Distinct?odhtype=odhmetadata&fields=Type&getasarray=true ``` After creation, tags are available for use in filtering via the **Content API**: * curl ``` curl "https://tourism.api.opendatahub.com/v1/Event?tagfilter=hiking" ``` info Tags created in the Data Browser become part of the shared vocabulary used by the **Content API** for filtering and classification. ## 4. Publishing news articles[​](#4-publishing-news-articles "Direct link to 4. Publishing news articles") News articles published via the Data Browser are consumed through the **Content API** under the `Article` entity type. Proper metadata ensures correct display in target applications such as the NOI Community App. ### 4.1. Required article properties[​](#41-required-article-properties "Direct link to 4.1. Required article properties") To publish an article: * Set `MainType` to `newsfeednoi` * Define `PublishedOn` to include `NOI Community App` * Set `Highlight` to `true` for top placement * Provide start and end dates for visibility * Enter multilingual content using the language selector Language-specific fields (`Title`, `BaseText`, etc.) are stored in the `TextInformation` object and returned by the **Content API** based on the requested language. ### 4.2. Contact and logo information[​](#42-contact-and-logo-information "Direct link to 4.2. Contact and logo information") Articles must include contact details: * Name: e.g., `NOI Techpark` * Email: e.g., `community@noi.bz.it` * Web URL * Logo: use one of the predefined image URLs hosted on `images.opendatahub.com` Example logo URL: ``` https://images.opendatahub.com/api/Image/GetImage?imageurl=NOI.png ``` info The **Content API** returns article data including `TextInformation`, `Contact`, and `Images`, enabling clients to render news items with full context. ## Related[​](#related "Direct link to Related") * [Data Browser overview](/tools/data-browser/.md) * [Content API reference](/use-data/content-api/reference.md) * [Filtering and sorting](/use-data/content-api/filtering-and-sorting.md) --- # Discovery [Open Data Hub Discovery](https://discovery.opendatahub.com) is a web tool for **exploring and understanding the datasets** in the Open Data Hub before you query them. It automatically discovers datasets from the Metadata API, infers their schema, and computes field-level statistics, so you can find the right dataset and the right fields without writing any code. It is the natural first step of a typical workflow: discover a dataset here, inspect it in the [Data Browser](/tools/data-browser/.md) or [Analytics](/tools/analytics.md), then read it through the [Content API](/use-data/content-api/reference.md) or [Time Series API](/use-data/time-series-api/reference.md). ## 1. What you can do[​](#1-what-you-can-do "Direct link to 1. What you can do") * **Search datasets** by keyword, tag, or domain (Mobility, Tourism). * **Inspect a dataset's schema**: the full list of fields with their paths and types. * **Read field-level statistics**: how often a field is populated (null percentage), how many distinct values it has, and its most common values, so you can tell whether a field is actually usable before building a request. * **Browse tags** used across datasets to understand how content is classified. ## 2. AI chatbot assistant[​](#2-ai-chatbot-assistant "Direct link to 2. AI chatbot assistant") Discovery includes an **AI chatbot assistant** that lets you explore datasets in natural language: ask which datasets exist, what fields they expose, or which values a field contains, and it answers using the Discovery data directly. info Access to the chatbot can be arranged on request. Contact the Open Data Hub team at `help@opendatahub.com` to enable it for your use. ## 3. From discovery to a request[​](#3-from-discovery-to-a-request "Direct link to 3. From discovery to a request") Once you have found a dataset and the fields you need, read it through the APIs: * Tourism content (accommodations, events, points of interest): [Content API reference](/use-data/content-api/reference.md). * Mobility time series (stations and measurements): [Time Series API reference](/use-data/time-series-api/reference.md). Each API also offers an interactive Swagger UI for trying requests directly in the browser. ## 4. Related[​](#4-related "Direct link to 4. Related") * [Quickstart: your first request](/quickstart.md) * [Data Browser](/tools/data-browser/.md) and [Analytics](/tools/analytics.md) * [Content API reference](/use-data/content-api/reference.md) and [Time Series API reference](/use-data/time-series-api/reference.md) --- # Imageresizer ## 1. Image Proxy Functionality[​](#1-image-proxy-functionality "Direct link to 1. Image Proxy Functionality") The Open Data Hub provides an image proxy service through the Content API to securely retrieve images hosted on HTTP endpoints and deliver them over HTTPS. This functionality is useful when consuming images from non-secure sources that must be displayed in secure contexts. The proxy endpoint is: ``` https://images.opendatahub.com/api/Image/GetImageByUrl ``` To retrieve an image, pass the source URL as the `imageurl` parameter. warning The Image Proxy only accepts the `imageurl` parameter. Additional query parameters are not supported unless they are handled by the target server. ### 1.1. Retrieve an image via proxy[​](#11-retrieve-an-image-via-proxy "Direct link to 1.1. Retrieve an image via proxy") * curl ``` curl "https://images.opendatahub.com/api/Image/GetImageByUrl?imageurl=http://daten.buergernetz.bz.it/services/weather/graphics/icons/imgsource/wetter/icon_2.png" -o weather-icon.png ``` This request fetches the image from the given HTTP URL and serves it over HTTPS. You can use this method to embed external images securely in web applications or data visualizations. ## 2. Image Resizing Service[​](#2-image-resizing-service "Direct link to 2. Image Resizing Service") The Content API supports dynamic image resizing for images hosted on specific Open Data Hub domains. This feature allows clients to request appropriately sized images without downloading full-resolution assets. ### 2.1. Supported domains[​](#21-supported-domains "Direct link to 2.1. Supported domains") Image resizing is available only for images hosted on the following domains: * `https://tourism.images.opendatahub.com` * `https://images.opendatahub.com` * `https://images.tourism.testingmachine.eu` warning You cannot resize external images directly. To resize an image from an external source, first retrieve it via the Image Proxy, then host it on a supported domain. ### 2.2. Resizing parameters[​](#22-resizing-parameters "Direct link to 2.2. Resizing parameters") To resize an image, append either `width` or `height` (or both) as query parameters to the image URL. | Parameter | Type | Required | Default | Description | | --------- | ------- | -------- | -------- | ------------------------ | | `width` | integer | no | original | Desired width in pixels | | `height` | integer | no | original | Desired height in pixels | * curl ``` curl "https://tourism.images.opendatahub.com/path/to/image.jpg?width=300" -o resized-image.jpg ``` ### 2.3. Aspect ratio behavior[​](#23-aspect-ratio-behavior "Direct link to 2.3. Aspect ratio behavior") * If you specify only `width` or only `height`, the image is scaled proportionally. * If you specify both `width` and `height`, the image is resized to the exact dimensions, which may result in distortion if the aspect ratio does not match the original. For example: ``` https://tourism.images.opendatahub.com/path/to/image.jpg?width=300&height=200 ``` Use proportional values to avoid unintended stretching. ## 3. Use cases for API users and ingestion creators[​](#3-use-cases-for-api-users-and-ingestion-creators "Direct link to 3. Use cases for API users and ingestion creators") * **API users** can leverage the image proxy to securely display weather icons or other HTTP-hosted media in HTTPS applications. * **Ingestion creators** can use the resizing service to generate thumbnails or optimized assets for downstream consumers by referencing hosted images with size constraints. For more information on consuming data, see [Content API Reference](/use-data/content-api/reference.md). --- # Web Components The Open Data Hub provides a collection of reusable Web Components for integrating real-time data into websites and applications. These components, covering domains such as tourism, weather, and mobility, are designed to be easily embedded and configured without requiring deep integration with the underlying APIs. You can browse and use the available Web Components at . Each component includes a "Share" button that provides an embeddable code snippet for quick integration. ## 1. Implementing a Web Component[​](#1-implementing-a-web-component "Direct link to 1. Implementing a Web Component") To use a Web Component from the Open Data Hub, follow these steps: 1. Visit the [Web Components library](https://webcomponents.opendatahub.com/). 2. Select the component you want to use. 3. Click the "Share" button to copy the embed code. 4. Paste the code into your HTML page. info Always place the Web Component code inside a `
` element to ensure proper rendering. For example, to embed the `day-trip-map-widget`: ```
``` Each component accepts its own attributes, set on the HTML tag. For the `day-trip-map-widget`, for example: | Parameter | Type | Required | Default | Description | | ----------------- | ------ | -------- | ------- | ------------------------------------------------------ | | `lang-and-locale` | string | no | en-US | Sets the language and locale (e.g., `de-DE`, `it-IT`). | | `log-info` | string | no | false | Enables logging output when set to `true`. | Refer to each component's own page in the [Web Components library](https://webcomponents.opendatahub.com/) for its full list of attributes. ## 2. Character encoding and emoji support[​](#2-character-encoding-and-emoji-support "Direct link to 2. Character encoding and emoji support") Some Web Components, such as `day-trip-map-widget`, use emojis or special icons that require UTF-8 encoding to render correctly. warning Without UTF-8 encoding, emojis and special characters may appear as boxes or question marks. Ensure your HTML document includes the following in the `` section: ``` ``` This applies to all Web Components that display weather icons, trail symbols, or other Unicode glyphs. ## Related[​](#related "Direct link to Related") * [Content API reference](/use-data/content-api/reference.md) * [Time Series API reference](/use-data/time-series-api/reference.md) --- # Authentication and authorization To access protected data in the Open Data Hub APIs, you must authenticate using OAuth 2.0 through Keycloak and include a valid access token in your API requests. This guide explains how to obtain and use tokens, manage their lifecycle, and request access to closed datasets. All instructions are written for API users who need to consume data from the Content API or Time Series API. ## 1. Getting started with authentication[​](#1-getting-started-with-authentication "Direct link to 1. Getting started with authentication") The Open Data Hub uses Keycloak as its identity provider for API access. Authentication is required to retrieve closed data or perform write operations. You will interact with the Keycloak server in the `noi` realm to obtain time-limited access tokens. You must use one of the supported OAuth 2.0 flows depending on your use case: * **Client Credentials flow**: for machine-to-machine (service) authentication. * **Password flow**: for user-based authentication using username and password. * **Refresh Token flow**: to renew an expired access token without re-entering credentials. The Content API and Time Series API both rely on these tokens but may differ in how they handle invalid or missing tokens. ### 1.1. Authentication endpoints[​](#11-authentication-endpoints "Direct link to 1.1. Authentication endpoints") Use the following token endpoint to request access tokens: ``` https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token ``` For production environments, use: ``` https://auth.opendatahub.com/auth/realms/noi/protocol/openid-connect/token ``` Always verify which environment (testing or production) your application targets. warning Production client credentials are not public. Contact Open Data Hub Support to obtain them. Development credentials are provided only for testing. ## 2. Using OAuth 2.0 flows[​](#2-using-oauth-20-flows "Direct link to 2. Using OAuth 2.0 flows") ### 2.1. Client Credentials flow (machine-to-machine)[​](#21-client-credentials-flow-machine-to-machine "Direct link to 2.1. Client Credentials flow (machine-to-machine)") Use this flow when your application runs as a background service or data collector that does not represent a user. This flow returns an `access_token` but no `refresh_token`. After the token expires, you must request a new one. #### Example: Obtain a token using client credentials[​](#example-obtain-a-token-using-client-credentials "Direct link to Example: Obtain a token using client credentials") * curl * Go ``` curl -X POST \ https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'grant_type=client_credentials&client_id=odh-mobility-datacollector-development&client_secret=7bd46f8f-c296-416d-a13d-dc81e68d0830&scope=openid' ``` ``` package main import ( "bytes" "fmt" "net/http" ) func main() { data := "grant_type=client_credentials&client_id=odh-mobility-datacollector-development&client_secret=7bd46f8f-c296-416d-a13d-dc81e68d0830&scope=openid" resp, _ := http.Post("https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token", "application/x-www-form-urlencoded", bytes.NewBuffer([]byte(data))) defer resp.Body.Close() fmt.Println("Status:", resp.Status) } ``` You will receive a JSON response containing the `access_token`: ``` { "access_token": "eyJhbGciOiJSUzI1NiIs...", "expires_in": 300, "refresh_expires_in": 0, "token_type": "Bearer" } ``` ### 2.2. Password flow (user-based login)[​](#22-password-flow-user-based-login "Direct link to 2.2. Password flow (user-based login)") Use this flow when you have a user account and want to authenticate directly with username and password. This flow is less secure than others and should be used only when necessary. #### Example: Get a token using username and password[​](#example-get-a-token-using-username-and-password "Direct link to Example: Get a token using username and password") * curl * Python ``` curl -X POST \ https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'grant_type=password&client_id=odh-generic-client&username=&password=&scope=openid' ``` ``` import requests url = "https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token" data = { "grant_type": "password", "client_id": "odh-generic-client", "username": "", "password": "", "scope": "openid" } response = requests.post(url, data=data) print(response.json()) ``` The response includes both an `access_token` and a `refresh_token`. ### 2.3. Refresh Token flow[​](#23-refresh-token-flow "Direct link to 2.3. Refresh Token flow") Use this flow to obtain a new `access_token` after it has expired, without requiring the user to re-enter credentials. #### Example: Refresh an expired access token[​](#example-refresh-an-expired-access-token "Direct link to Example: Refresh an expired access token") * curl ``` curl -X POST \ https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'grant_type=refresh_token&client_id=odh-generic-client&client_secret=&refresh_token=&scope=openid' ``` ## 3. Sending the access token[​](#3-sending-the-access-token "Direct link to 3. Sending the access token") Once you have obtained an access token, include it in the `Authorization` header of every API request as a Bearer token. ### 3.1. Format of the Authorization header[​](#31-format-of-the-authorization-header "Direct link to 3.1. Format of the Authorization header") ``` Authorization: Bearer ``` #### Example: Call the Content API with a token[​](#example-call-the-content-api-with-a-token "Direct link to Example: Call the Content API with a token") * curl * JavaScript ``` curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi" \ -H "Authorization: Bearer $TOKEN" ``` ``` fetch('https://tourism.api.opendatahub.com/v1/ODHActivityPoi', { headers: { 'Authorization': 'Bearer ' + token } }) .then(response => response.json()) .then(data => console.log(data)); ``` ## 4. Token lifetime and renewal[​](#4-token-lifetime-and-renewal "Direct link to 4. Token lifetime and renewal") Tokens have limited validity. You must handle expiration and refresh tokens accordingly. | Token Type | Lifetime | Notes | | --------------- | ---------------------------- | ------------------------------------- | | `access_token` | 300 seconds (5 minutes) | Must be refreshed after expiry. | | `refresh_token` | Up to 7200 seconds (2 hours) | Depends on Keycloak session settings. | warning If your token expires, the Content API may return a `200 OK` response but omit closed data fields. The Time Series API may silently filter out restricted records. Always check token validity before making requests. The `refresh_token` lifetime is determined by the minimum of: * SSO Session Max * SSO Session Idle * Client-specific session limits (if configured) Re-authenticate when both tokens have expired. ## 5. Accessing closed datasets[​](#5-accessing-closed-datasets "Direct link to 5. Accessing closed datasets") Some datasets are closed and require specific roles for access. These roles are managed via Keycloak and enforced by the Time Series API and Content API using role-based access control (RBAC). ### 5.1. Understanding roles[​](#51-understanding-roles "Direct link to 5.1. Understanding roles") * Roles prefixed with `BDP_` (e.g., `BDP_VIEW_CLOSED_DATA`) grant access to closed datasets. * The `GUEST` role defines what data is publicly available. * `ODH_ROLE` roles manage quotas or billing plans, not data access. To access closed data, you must be assigned a `BDP_` role in Keycloak. ### 5.2. Requesting access[​](#52-requesting-access "Direct link to 5.2. Requesting access") If you need access to a closed dataset: 1. Contact Open Data Hub Support and specify: * Your user account (email) * The dataset or origin you need access to (e.g., Laimburg sensor data) 2. The team will assign the appropriate `BDP_` role to your account in Keycloak. You do not need to manage ACL rule files or modify configuration repositories — this is handled by administrators. ### 5.3. Verifying role assignment[​](#53-verifying-role-assignment "Direct link to 5.3. Verifying role assignment") After being granted a role, ensure your client application receives the correct scope: * If `Full Scope Allowed` is enabled on the client, all roles are included automatically. * If disabled, specific roles must be manually assigned under **Client Roles** in Keycloak. Without proper scoping, your token will not include the required roles even if they are assigned to your user. ## 6. Tools and interfaces[​](#6-tools-and-interfaces "Direct link to 6. Tools and interfaces") You can manage authentication through various tools. ### 6.1. Swagger UI[​](#61-swagger-ui "Direct link to 6.1. Swagger UI") In Swagger interfaces (e.g., for the Content API or Time Series API): 1. Click **Authorize**. 2. Enter your username, password, and `client_id` (e.g., `odh-generic-client`). 3. Click **Authorize** to apply the token to all operations. Swagger stores the token in your browser session. warning To log out, click **Logout** in the Authorize dialog. Closing the tab may not invalidate the session. ### 6.2. Postman[​](#62-postman "Direct link to 6.2. Postman") In Postman: 1. Go to the **Authorization** tab. 2. Select **OAuth 2.0**. 3. Choose **Password Credentials** grant type. 4. Enter: * Token Name * Auth URL (not used for password flow) * Access Token URL: `https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token` * Client ID: `odh-generic-client` * Username and Password * Scope: `openid` 5. Click **Get New Access Token**, then **Use Token**. Ensure your environment variables or headers are set correctly to avoid sending unauthenticated requests. ## 7. Common issues and troubleshooting[​](#7-common-issues-and-troubleshooting "Direct link to 7. Common issues and troubleshooting") | Issue | Cause | Solution | | ----------------------------------- | ------------------------------------ | ----------------------------------------------------------- | | `401 Unauthorized` | Invalid, malformed, or expired token | Request a new token | | No closed data returned | Token missing required `BDP_` role | Contact support to request role assignment | | `context deadline exceeded` in logs | Inactive telemetry system | This can be safely ignored | | Multiple accounts with same email | Duplicate identities in Keycloak | Agree to combine accounts and follow on-screen instructions | info The `odh-generic-client` can be used without a `client_secret` for development. Use any string as `client_id`, such as `My test request`. warning Never hardcode production credentials in client applications. Use secure secret management practices. --- # Quotas, CLI and the R package ## 1. API quotas and rate limiting[​](#1-api-quotas-and-rate-limiting "Direct link to 1. API quotas and rate limiting") The Open Data Hub enforces quotas and rate limits to ensure fair usage and system stability. These limits vary based on your user type and affect both the historical data you can access and the number of requests you can make per second. The specific behavior described here applies to the **Time Series API** and **Content API**. ### 1.1. Quota limits by user type[​](#11-quota-limits-by-user-type "Direct link to 1.1. Quota limits by user type") Your access level determines your quota for historical data and request frequency. The following table summarizes the limits: | User Type | Max historical data | Requests per second | Rate limiting criteria | | ---------------------------- | ------------------- | ------------------- | ---------------------------------------- | | Unregistered (no referrer) | 5 days | 10 | per IP and query path | | Unregistered (with referrer) | 100 days | 20 | per referer, IP, and query path | | Basic | 1,000 days | 50 | per user-id, referer, IP, and query path | | Advanced | unlimited | 100 | per user-id, referer, IP, and query path | | Premium | unlimited | 200 | per user-id, referer, IP, and query path | warning Unregistered users can significantly increase their historical data access from 5 to 100 days by providing a `Referer` header or `referer` query parameter. Always include this identifier to benefit from higher limits. ### 1.2. Handling quota exhaustion[​](#12-handling-quota-exhaustion "Direct link to 1.2. Handling quota exhaustion") If you exceed your quota, the API returns an HTTP `429 Too Many Requests` error. The response body includes a message and a hint for further information. Example `429` response: ``` { "hint": "https://github.com/noi-techpark/odh-docs/wiki/Api-Quota", "message": "You have exhausted your API Request Quota" } ``` warning The `hint` URL in the error response points to a legacy wiki page. For up-to-date information, refer to the current Historical Data and Request Rate Limits documentation. ## 2. Using the HTTP Referer[​](#2-using-the-http-referer "Direct link to 2. Using the HTTP Referer") The **Time Series API** and **Content API** require the HTTP `Referer` header to support usage statistics, problem tracing, and improved rate limiting. Providing this header helps the Open Data Hub team understand how the APIs are used and assists in debugging issues. ### 2.1. Setting the Referer header[​](#21-setting-the-referer-header "Direct link to 2.1. Setting the Referer header") The `Referer` header should contain the URL of the application or webpage making the request. Browsers typically set this automatically, but you may need to set it manually in scripts or applications. * curl ``` curl -H 'Referer: https://myapp.bz.it/weather' \ 'https://api.opendatahub.com/v1/stations' ``` ### 2.2. Using the referer query parameter[​](#22-using-the-referer-query-parameter "Direct link to 2.2. Using the referer query parameter") If you cannot set HTTP headers, use the `referer` query parameter as a fallback. The API gives precedence to the query parameter over the header if both are present. The value should be a unique identifier that describes your application. * curl ``` curl 'https://api.opendatahub.com/v1/stations?referer=myapp-unique-id' ``` info Always prefer setting the `Referer` header. Use the query parameter only when technical constraints prevent header usage. A meaningful identifier improves support and analytics. ## 3. Command line access with curl[​](#3-command-line-access-with-curl "Direct link to 3. Command line access with curl") The **Time Series API** and **Content API** support command-line access via tools like `curl`, enabling non-interactive, scriptable data retrieval. This is useful for automation, data processing, and integration with third-party systems. ### 3.1. Retrieving data with curl[​](#31-retrieving-data-with-curl "Direct link to 3.1. Retrieving data with curl") You can use `curl` to fetch data directly from the API. The Swagger interfaces for the APIs provide ready-to-use `curl` commands for every query. * curl ``` curl -H 'Referer: https://myapp.bz.it' \ 'https://mobility.api.opendatahub.com/v2/stations' ``` ### 3.2. Alternative tools[​](#32-alternative-tools "Direct link to 3.2. Alternative tools") While `curl` is the most common tool, you can use alternatives like `wget`. Ensure you adapt the parameters correctly for the tool you choose. Example with `wget`: ``` wget --header='Referer: https://myapp.bz.it' \ 'https://api.opendatahub.com/v1/stations' ``` info Command-line access is ideal for one-way data retrieval in scripts, automated updates, or feeding data into applications like kiosk displays. It is not suited for interactive workflows. ## 4. Accessing BZ Analytics data with the bzar R package[​](#4-accessing-bz-analytics-data-with-the-bzar-r-package "Direct link to 4. Accessing BZ Analytics data with the bzar R package") The **bzar** R package, developed by the Open Data Hub team, provides an interface to **BZ Analytics** data. It allows R users to retrieve, analyze, and visualize data using R's extensive statistical and graphics capabilities. ### 4.1. Installing the bzar package[​](#41-installing-the-bzar-package "Direct link to 4.1. Installing the bzar package") The package is hosted on GitHub and can be installed using the `devtools` package. * R ``` # Install devtools if not already installed # install.packages("devtools") devtools::install_github('noi-techpark/it.bz.opendatahub.analytics.libs', subdir='api/R') ``` warning You must have R and the `devtools` package installed to use this method. The package is under active development, so functionality and API stability may change. ### 4.2. Retrieving and visualizing data[​](#42-retrieving-and-visualizing-data "Direct link to 4.2. Retrieving and visualizing data") After installation, use the `bz_data()` function to query BZ Analytics data. The result can be processed with standard R functions or visualized using `ggplot2`. * R ``` library(bzar) # Retrieve data result <- bz_data(query = 'your_query') # Basic plot plot(result$timestamp, result$value) # Or use ggplot2 library(ggplot2) ggplot(result, aes(x = timestamp, y = value)) + geom_line() ``` info The `bzar` package is specifically designed for BZ Analytics data and may require authentication for certain endpoints. Refer to the [bzar repository](https://github.com/noi-techpark/it.bz.opendatahub.analytics.libs/tree/main/api/R) for detailed documentation and examples. **Related**: [Authentication and authorization](/use-data/authentication-and-access/authentication.md), [Time Series API reference](/use-data/time-series-api/reference.md) --- # AlpineBits integration The Content API supports integration with the AlpineBits standard, enabling access to tourism data through standardized endpoints. Open Data Hub implements two key datasets defined by the AlpineBits Alliance: HotelData and DestinationData. These datasets are served via separate endpoints and require specific request formatting to retrieve data successfully. ## 1. AlpineBits endpoints in the Content API[​](#1-alpinebits-endpoints-in-the-content-api "Direct link to 1. AlpineBits endpoints in the Content API") The Content API exposes two distinct AlpineBits endpoints, each dedicated to a specific dataset. Ensure you use the correct endpoint and parameters for your use case. ### 1.1. HotelData endpoint[​](#11-hoteldata-endpoint "Direct link to 1.1. HotelData endpoint") The HotelData dataset provides information related to accommodations and booking, such as room inventory and availability. Access this dataset via the following endpoint: * **URL**: `https://alpinebits.opendatahub.com/AlpineBits` This endpoint supports actions like `FreeRooms`, `Inventory Basic`, and `Inventory HotelInfo`, depending on the protocol version used. ### 1.2. DestinationData endpoint[​](#12-destinationdata-endpoint "Direct link to 1.2. DestinationData endpoint") The DestinationData dataset offers structured information about tourism-related entities such as events, mountain areas, trails, and places. The Open Data Hub hosts the reference implementation of the AlpineBits DestinationData standard. * **URL**: `https://destinationdata.alpinebits.opendatahub.com/` DestinationData is built using modern web standards including JSON, REST API, Schema.org, and OntoUML. Its core output is the AlpineBits DestinationData Ontology, which defines named entities for consistent data modeling. info For full details on the DestinationData Ontology and API architecture, refer to the [official DestinationData specification](https://www.alpinebits.org/wp-content/uploads/2021/05/AlpineBits-DestinationData-2021-04.pdf). ## 2. Request requirements[​](#2-request-requirements "Direct link to 2. Request requirements") To retrieve data from either AlpineBits endpoint via the Content API, your requests must meet specific requirements. ### 2.1. HTTP method and content type[​](#21-http-method-and-content-type "Direct link to 2.1. HTTP method and content type") All requests must: * Use the **POST** method * Set the `Content-Type` header to `multipart/form-data` ### 2.2. Required headers[​](#22-required-headers "Direct link to 2.2. Required headers") Include the following headers in every request: | Header | Description | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Authorization` | Use Basic authentication. While not strictly required, provide your email address as the token to aid debugging and support. Example: `Basic user@example.com` | | `X-AlpineBits-ClientProtocolVersion` | Specify a supported protocol version, such as `2017-10` or `2018-10` | | `X-AlpineBits-ClientID` | A descriptive identifier for your client or request, e.g., `My test request` | ### 2.3. Action parameter[​](#23-action-parameter "Direct link to 2.3. Action parameter") You must include an `action` parameter in the form data to specify the operation. Valid actions depend on the protocol version and dataset. For example: * `action=getVersion` retrieves the server version * `action=FreeRooms` queries available rooms (HotelData only) See the protocol matrix in the source documentation for full action support by version. ## 3. Example request[​](#3-example-request "Direct link to 3. Example request") The following example demonstrates how to query the HotelData endpoint using the `getVersion` action. * curl ``` curl --location --request POST 'https://alpinebits.opendatahub.com/AlpineBits' \ --header 'Authorization: Basic user@example.com' \ --header 'X-AlpineBits-ClientProtocolVersion: 2017-10' \ --header 'X-AlpineBits-ClientID: My test request' \ --header 'Content-Type: multipart/form-data' \ --form 'action=getVersion' ``` warning Ensure you use the correct action and headers. Mismatched protocol versions or missing parameters may result in empty or unauthorized responses. ## 4. Additional resources[​](#4-additional-resources "Direct link to 4. Additional resources") * [AlpineBits Alliance official website](https://www.alpinebits.org/) * [DestinationData documentation](https://www.alpinebits.org/destinationdata/) * [HotelData documentation](https://www.alpinebits.org/hoteldata/) Related: Query the Content API --- # Deprecations and datamodel migration ## 1. Introduction[​](#1-introduction "Direct link to 1. Introduction") The Content API regularly updates its data models to improve consistency, performance, and alignment with tagging standards. As part of this evolution, certain fields and endpoints are deprecated or removed. This page documents all deprecated and removed fields, structural model changes, and migration guidance specific to the Content API. If you consume data from the Content API, review this page to ensure your integrations remain compatible. warning Using deprecated fields or endpoints is discouraged. Deprecated elements may be removed after their `RemovedAfter` date, which can break existing integrations. Always migrate to recommended replacements as soon as possible. ## 2. Deprecated and removed fields[​](#2-deprecated-and-removed-fields "Direct link to 2. Deprecated and removed fields") The Content API has deprecated or removed several fields across key data models. These changes streamline data structure and promote the use of standardized tagging and metadata fields. ### 2.1. Events datamodel changes[​](#21-events-datamodel-changes "Direct link to 2.1. Events datamodel changes") The following fields have been removed from the Events datamodel and are no longer available in the JSON response: * `Pdf`, `Ranc`, `PayMet`, `Type`, `GrpEvent` * `LTSTags`, `Hashtag` * `EventDate.GpsEast`, `EventDate.GpsNorth`, `EventDate.InscriptionTill`, `EventDate.EventDateAdditionalTime` * `EventPrices`, `EventBenefit`, `EventCrossSelling`, `EventDescAdditional`, `EventOperationScheduleOverview`, `NextBeginDate` The following fields are deprecated and will be removed in a future version. Transition to the recommended replacements: | Deprecated Field | Replacement | Notes | | ----------------------------------------------------------------------- | ------------------------------------ | ----------------------------------- | | `EventDate.EventCalculatedDay` | `EventDate.EventCalculatedDays` | Now returned as a list | | `EventPrice` | `EventVariants` | Use variant-based pricing | | `ODHTags` | `Tags` | Use the unified `Tags` field | | `OdhActive` | `PublishedOn` | Indicates publication status | | `EventBooking.BookableTo`, `BookableFrom`, `AccommodationAssignment` | `EventUrls` (type `bookingUrl`) | Booking URLs are now in `EventUrls` | | `ClassificationRID` | `Tags` (type `eventclassification`) | Migrated to tagging system | | `Topics`, `TopicRids` | `Tags` (type `eventcategory`) | Migrated to tagging system | | `Gpstype`, `Latitude`, `Longitude`, `Altitude`, `AltitudeUnitofMeasure` | `GpsInfo` | Use the structured GPS object | | `Ticket` | `EventProperty.TicketRequired` | Moved to property object | | `SignOn` | `EventProperty.RegistrationRequired` | Moved to property object | | `OrgRID` | `EventProperty.EventOrganizerId` | Moved to property object | | `EventAdditionalInfos.Mplace` | `MeetingPoint` | Updated field name | | `EventAdditionalInfos.Reg` | `Registration` | Updated field name | ### 2.2. ODHActivityPoi datamodel changes[​](#22-odhactivitypoi-datamodel-changes "Direct link to 2.2. ODHActivityPoi datamodel changes") The following fields will be removed from the ODHActivityPoi datamodel: * `CustomId`, `SmgId`, `Type`, `SubType`, `PoiType` * `AdditionalPoiInfos.MainType`, `AdditionalPoiInfos.SubType`, `AdditionalPoiInfos.PoiType` * `Highlight`, `OwnerRid`, `ChildPoiIds`, `MasterPoiIds`, `PoiServices`, `ODHActivityPoiTypes` * `LocationInfo.AreaInfo` The following fields are deprecated and will be removed. Use the replacements listed: | Deprecated Field | Replacement | Notes | | -------------------------------------------------------------- | ------------- | ---------------------------------------- | | `PoiProperty` | - | Obsolete, no replacement | | `SyncSourceInterface`, `SyncUpdateMode` | - | Obsolete, no replacement | | `GpsPoints` | `GpsInfo` | Use the structured GPS object | | `OutdooractiveID`, `OutdooractiveElevationID` | `Mapping` | May be moved to `Mapping` field | | `Difficulty` | `Ratings` | Use the corresponding field in `Ratings` | | `LTSTags`, `SmgTags` | `Tags` | Use the unified `Tags` field | | `SmgActive` | `PublishedOn` | Indicates publication status | | `CustomId`, `SmgId`, `OwnerRid`, `ChildPoiIds`, `MasterPoiIds` | `Mapping` | May be preserved in `Mapping` | info The `GpsPoints` field is deprecated and should be replaced with `GpsInfo`. Although some sources previously suggested the reverse, the correct migration path is to use `GpsInfo` and phase out `GpsPoints`. ### 2.3. General field replacements[​](#23-general-field-replacements "Direct link to 2.3. General field replacements") Across multiple datamodels, the following deprecated fields should be replaced as indicated: * `LTSTags`, `SmgTags`, `Type` → use `Tags` * `GpsPoints` → use `GpsInfo` * `LocationInfo.AreaInfo` → use the `AreaIds` array * `MainEntity` → use `ValidForEntity` * `IDMCategoryMapping` → obsolete, do not use * `LTSTaggingInfo` → replaced by the `Mapping` field ## 3. Viewing deprecation status[​](#3-viewing-deprecation-status "Direct link to 3. Viewing deprecation status") The Content API provides multiple ways to identify deprecated fields and endpoints. ### 3.1. Deprecated endpoint (deprecated)[​](#31-deprecated-endpoint-deprecated "Direct link to 3.1. Deprecated endpoint (deprecated)") The Content API previously exposed a route to list all deprecated fields: * curl * Go * Python * JavaScript ``` curl "https://tourism.api.opendatahub.com/v1/Deprecated" -H "Authorization: Bearer $TOKEN" ``` ``` // Example: Fetch deprecated fields resp, _ := http.Get("https://tourism.api.opendatahub.com/v1/Deprecated") ``` ``` import requests url = "https://tourism.api.opendatahub.com/v1/Deprecated" headers = {"Authorization": "Bearer "} response = requests.get(url, headers=headers) ``` ``` fetch('https://tourism.api.opendatahub.com/v1/Deprecated', { headers: { 'Authorization': 'Bearer ' } }) ``` Example response: ``` { "weatherdistrict": [ { "Name": "date", "Type": "DateTime", "Description": "Obsolete, use Date instead", "DeprecationDate": "2022-01-01", "RemovedAfter": "2023-01-01" } ] } ``` warning The `/v1/Deprecated` endpoint is itself deprecated and may be removed in a future version. Rely on schema annotations and the Data Browser for up-to-date deprecation status. ### 3.2. API schemas[​](#32-api-schemas "Direct link to 3.2. API schemas") In the Content API Swagger documentation, deprecated fields and endpoints are marked with a visual indicator. You can inspect the schema definitions directly to see deprecation metadata such as `DeprecationDate` and `RemovedAfter`. ### 3.3. Data Browser[​](#33-data-browser "Direct link to 3.3. Data Browser") In the [Open Data Hub Data Browser](https://databrowser.opendatahub.com), deprecated fields are visually marked with an orange dot or outline. Clicking the marker reveals additional information, including the reason for deprecation and the recommended replacement. info Deprecation markers in the Data Browser require user interaction to view details. Regularly check the Data Browser to stay informed about upcoming changes. ## 4. Data migration and structural model changes[​](#4-data-migration-and-structural-model-changes "Direct link to 4. Data migration and structural model changes") The Open Data Hub is transitioning from a Legacy Instance (`service.suedtirol.info`) to the Main Instance (`tourism.api.opendatahub.com`). Most data types have been fully migrated. ### 4.1. Fully migrated data types[​](#41-fully-migrated-data-types "Direct link to 4.1. Fully migrated data types") The following data types are fully migrated to the Main Instance and should be edited there: * Event, Gastronomy, Activity, Poi, Accommodation * Webcams, Measuringpoints, Venues, ODHTags * District, Municipality, TV, Region * Skiarea/Skiregion, Wine Companies, Availability Search ### 4.2. Partially migrated data types[​](#42-partially-migrated-data-types "Direct link to 4.2. Partially migrated data types") * **Snow Report**: Partially migrated. Some functionality may still reside on the Legacy Instance, but all data can now be edited on the Main Instance. info If Save/Delete functionality is unavailable on the Legacy Instance, you must edit the data on the Main Instance. ### 4.3. Structural changes: AdditionalProperties[​](#43-structural-changes-additionalproperties "Direct link to 4.3. Structural changes: AdditionalProperties") For LTS-sourced data types (Gastronomy, POI, Activity), several root-level properties are being moved into the `AdditionalProperties` object to improve data organization. These changes are backward-compatible for several months, but you should update your code to access the new locations. #### Gastronomy (LTS)[​](#gastronomy-lts "Direct link to Gastronomy (LTS)") The following fields are moving to `AdditionalProperties.GastronomyLtsDataProperties`: * `CategoryCodes`, `DishRates`, `CapacityCeremony`, `Facilities`, `MaxSeatingCapacity` #### POI (LTS)[​](#poi-lts "Direct link to POI (LTS)") * `AgeFrom`, `AgeTo` → `AdditionalProperties.PoiAgeDataProperties` * `IsOpen`, `HasFreeEntrance` → `AdditionalProperties.PoiLtsDataProperties` #### Activity (LTS)[​](#activity-lts "Direct link to Activity (LTS)") * `AltitudeDifference`, `AltitudeHighestPoint`, `AltitudeLowestPoint`, `AltitudeSumUp`, `AltitudeSumDown`, `DistanceDuration`, `DistanceLength`, `Highlight`, `IsPrepared`, `RunToValley`, `IsWithLigth`, `HasRentals`, `LiftAvailable`, `FeetClimb`, `BikeTransport`, `WayNumber`, `Number`, `IsOpen`, `HasFreeEntrance` ### 4.4. Supported schema.org types[​](#44-supported-schemaorg-types "Direct link to 4.4. Supported schema.org types") The Content API supports retrieving data in JSON-LD format using schema.org types. The following types are currently supported: * `Hotel`, `Restaurant`, `Event`, `TouristAttraction`, `Recipe`, `SkiResort`, `Place` To retrieve data in JSON-LD format, set the `Accept` header: ``` Accept: application/ld+json ``` ## 5. Related[​](#5-related "Direct link to 5. Related") * [Content API reference](/use-data/content-api/reference.md) * [Filtering and sorting (Content API)](/use-data/content-api/filtering-and-sorting.md) * [Data Browser](/tools/data-browser/.md) --- # Filtering and sorting (Content API) The Content API provides a comprehensive set of query parameters to filter, sort, and shape the data returned from its endpoints. These tools allow API users to retrieve only the data they need, reduce payload size, and organize results effectively. This guide covers the core filtering and sorting capabilities available across Content API endpoints. ## 1. Field selection and null value handling[​](#1-field-selection-and-null-value-handling "Direct link to 1. Field selection and null value handling") Control which fields are included in the response and manage null or empty values to optimize payload size and structure. ### 1.1 Selecting fields with `fields`[​](#11-selecting-fields-with-fields "Direct link to 11-selecting-fields-with-fields") Use the `fields` parameter to specify which fields to include in the response. The `Id` field is always included, even if not explicitly requested. You can select: * Top-level fields: `Shortname` * Nested object fields: `Detail.en.Title` * Array elements: `ODHTags.[0]`, `ODHTags.[*].Id` Multiple fields are comma-separated. | Parameter | Type | Required | Default | Description | | --------- | ------ | -------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `fields` | string | no | all fields | Comma-separated list of fields to include in the response. Supports dot notation for nested fields and bracket notation for arrays. | info Field names in the `fields` parameter are case-sensitive and must match the API schema exactly. * Go * curl * Python * JavaScript ``` // Fetch only Shortname and German title url := "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?fields=Shortname,Detail.de.Title" ``` ``` curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?fields=Shortname,Detail.de.Title" -H "Authorization: Bearer $TOKEN" ``` ``` import requests url = "https://tourism.api.opendatahub.com/v1/ODHActivityPoi" params = {"fields": "Shortname,Detail.de.Title"} headers = {"Authorization": "Bearer "} response = requests.get(url, params=params, headers=headers) ``` ``` const url = new URL('https://tourism.api.opendatahub.com/v1/ODHActivityPoi'); url.searchParams.append('fields', 'Shortname,Detail.de.Title'); fetch(url, { headers: { 'Authorization': 'Bearer ' } }); ``` ### 1.2 Handling null and empty values[​](#12-handling-null-and-empty-values "Direct link to 1.2 Handling null and empty values") Control how null and empty values appear in the response. | Parameter | Type | Required | Default | Description | | -------------------------- | ------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------- | | `removenullvalues` | boolean | no | false | If true, removes fields with `null` values from the JSON response. | | `excludenulloremptyvalues` | boolean | no | false | If true, removes fields with `null` or empty string values. Used primarily in the Distinct Search endpoint. | warning The `excludenulloremptyvalues` parameter is specific to certain endpoints like Distinct Search and may not affect all Content API responses. ## 2. Filtering data[​](#2-filtering-data "Direct link to 2. Filtering data") Apply filters to narrow down results based on tags, text search, location, or custom conditions. ### 2.1 Tag-based filtering with `tagfilter`[​](#21-tag-based-filtering-with-tagfilter "Direct link to 21-tag-based-filtering-with-tagfilter") Filter entries by associated tags using the `tagfilter` parameter. Tags can be from the Open Data Hub system or external sources. | Parameter | Type | Required | Default | Description | | ----------- | ------ | -------- | ------- | ---------------------------------------------------------------------------------------------------------------- | | `tagfilter` | string | no | - | Filters results by tag expression. Supports comma-separated values (OR logic) or explicit `and()`/`or()` syntax. | * Comma-separated (OR logic): `tagfilter=hiking,bike` * Explicit OR: `tagfilter=or(hiking,bike)` * AND logic: `tagfilter=and(hiking,summer)` * Source-specific: `tagfilter=idm.paragliding` info Tag values are case-sensitive identifiers in lowercase (for example `hiking`, not `Hiking`), not display labels. Use `tagfilter`; the `odhtagfilter` parameter does not match these values. Discover the valid tags for an entity with the [`/v1/Tag` endpoint](/use-data/content-api/reference.md#41-tags), for example `GET /v1/Tag?validforentity=odhactivitypoi`. A tag that does not exist returns zero results rather than an error. ### 2.2 Text search with `searchfilter`[​](#22-text-search-with-searchfilter "Direct link to 22-text-search-with-searchfilter") Search across title fields and IDs using the `searchfilter` parameter. | Parameter | Type | Required | Default | Description | | ---------------- | ------- | -------- | ------- | ------------------------------------------------------------------------------------------- | | `searchfilter` | string | no | - | Searches title fields and IDs. Use with `language` to scope search. | | `language` | string | no | en | Limits search to content in the specified language (e.g., `de`, `it`, `en`). | | `searchbasetext` | boolean | no | false | If true, extends search to base text fields. Can significantly degrade performance. | | `filteronfields` | string | no | - | Comma-separated list of nested field paths to search (e.g., `ContactInfos.en.CompanyName`). | warning Omitting the `language` parameter may result in excessive results and slower responses, as the search runs across all available languages. warning Using `searchbasetext=true` can significantly degrade API performance. Use sparingly. ### 2.3 Location-based filtering[​](#23-location-based-filtering "Direct link to 2.3 Location-based filtering") Filter results by geographic area using coordinates, polygons, or location identifiers. #### Radius-based filtering[​](#radius-based-filtering "Direct link to Radius-based filtering") Use `latitude`, `longitude`, and `radius` (in meters) to find items within a circular area. Results are automatically sorted by ascending distance. | Parameter | Type | Required | Default | Description | | ----------- | ------ | ---------------------------------- | ------- | --------------------------------------- | | `latitude` | number | yes (with `longitude`) | - | Decimal latitude of the search center. | | `longitude` | number | yes (with `latitude`) | - | Decimal longitude of the search center. | | `radius` | number | yes (with `latitude`, `longitude`) | - | Search radius in meters. | * curl ``` curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?latitude=46.624975&longitude=11.369909&radius=2000" -H "Authorization: Bearer $TOKEN" ``` #### Polygon filtering[​](#polygon-filtering "Direct link to Polygon filtering") Use the `polygon` parameter to filter by geographic area. Supports multiple formats: * **WKT**: `POLYGON((11.026805 46.688285, ...))` * **Custom syntax**: `bbc(...)` (bounding box closed), `bbi(...)` (bounding box intersected) * **GeoShapes API reference**: `it.municipality.Bolzano/Bozen` warning Large polygon definitions may exceed URL length limits for GET requests. Consider using POST with a body if this occurs. #### Location identifier filtering (`locfilter`)[​](#location-identifier-filtering-locfilter "Direct link to location-identifier-filtering-locfilter") Filter by predefined location types using `locfilter`. Combine multiple values with commas (OR logic). | Type prefix | Description | | ----------- | ------------------- | | `tvs` | Tourism association | | `mun` | Municipality | | `fra` | Fraction | Example: `locfilter=tvs522822D451CA11D18F1400A02427D15E,mun99A8B1D4A8D64303B1B965AA7C20FA60` ### 2.4 Custom filtering with `rawfilter`[​](#24-custom-filtering-with-rawfilter "Direct link to 24-custom-filtering-with-rawfilter") Apply advanced filtering logic using the `rawfilter` parameter with a functional syntax. | Function | Meaning | | ---------------------- | ------------------------------------- | | `eq(field, value)` | Equal | | `ne(field, value)` | Not equal | | `gt(field, value)` | Greater than | | `ge(field, value)` | Greater than or equal | | `lt(field, value)` | Less than | | `le(field, value)` | Less than or equal | | `and(...)` | Logical AND | | `or(...)` | Logical OR | | `isnull(field)` | Field is null | | `isnotnull(field)` | Field is not null | | `in(field, value)` | Value in array | | `nin(field, value)` | Value not in array | | `like(field, value)` | Case-insensitive substring match | | `likein(field, value)` | Substring match within array elements | Field paths support dot notation (`Detail.de.Title`) and array indexing (`ODHTags.[0].Id`, `ODHTags.[*].Id`). * curl ``` # Active hiking activities with non-null altitude curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?rawfilter=and(eq(Type,'Wandern'),isnotnull(Geo.0.Altitude))" -H "Authorization: Bearer $TOKEN" ``` warning Negation (e.g., `not`) is not supported. Use `ne` or `isnotnull` instead. ## 3. Sorting results[​](#3-sorting-results "Direct link to 3. Sorting results") Control the order of results using built-in or custom sorting. ### 3.1 Pagination and consistent sorting[​](#31-pagination-and-consistent-sorting "Direct link to 3.1 Pagination and consistent sorting") Use `pagesize` and `pagenumber` for pagination. The Content API does **not** use `limit`/`offset` (those belong to the Time Series API); supplying them has no effect. | Parameter | Type | Required | Default | Description | | ------------ | ------ | -------- | ------- | --------------------------------------------------------------------------------------- | | `pagesize` | number | no | 10 | Number of items per page (max 1000). | | `pagenumber` | number | no | 1 | Page number to retrieve (1-based). | | `seed` | number | no | null | Ensures stable ordering across pages. Use `seed=0` to get a random seed, then reuse it. | ### 3.2 Custom sorting with `rawsort`[​](#32-custom-sorting-with-rawsort "Direct link to 32-custom-sorting-with-rawsort") Sort results by any field using `rawsort`. Prefix with `-` for descending order. | Parameter | Type | Required | Default | Description | | --------- | ------ | -------- | ------- | -------------------------------------------------------------------------- | | `rawsort` | string | no | - | Comma-separated list of fields to sort by. Prefix with `-` for descending. | * curl ``` # Sort by German title ascending, then altitude descending curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?rawsort=Detail.de.Title,-Geo.0.Altitude" -H "Authorization: Bearer $TOKEN" ``` warning An active geofilter (e.g., `latitude`/`longitude`/`radius`) overrides the `rawsort` parameter, sorting results by distance instead. warning The `rawsort` parameter only works on fields that are included in the selected output (via `fields` or full projection). ## 4. Specialized endpoints[​](#4-specialized-endpoints "Direct link to 4. Specialized endpoints") ### 4.1 Distinct Search endpoint[​](#41-distinct-search-endpoint "Direct link to 4.1 Distinct Search endpoint") Retrieve unique values for specified fields using the Distinct Search endpoint. * **Endpoint**: `https://tourism.api.opendatahub.com/v1/Distinct` * **Required parameters**: * `type`: Data type to search (e.g., `Accommodation`, `Event`) * `fields`: Field(s) to retrieve distinct values for Supports `rawfilter`, `rawsort`, and `excludenulloremptyvalues`. * curl ``` # Get distinct sources for webcams curl "https://tourism.api.opendatahub.com/v1/Distinct?type=webcam&fields=Source" -H "Authorization: Bearer $TOKEN" ``` warning The Distinct Search endpoint is currently available on the test instance. Verify availability before using in production. info Multiple fields in `fields` are only supported for non-array fields. ## Related[​](#related "Direct link to Related") * [Content API reference](/use-data/content-api/reference.md) * [Output formats (Content API)](/use-data/content-api/output-formats.md) * [Authentication and authorization](/use-data/authentication-and-access/authentication.md) --- # Output formats (Content API) ## 1. JSON and JSON-LD response formats[​](#1-json-and-json-ld-response-formats "Direct link to 1. JSON and JSON-LD response formats") The Content API returns data primarily in JSON format. You can optionally receive responses in JSON-LD format, which models data using the schema.org vocabulary, by setting the appropriate HTTP header. To request JSON-LD output, set the `Accept` header to `application/ld+json`. The Content API will return structured data compatible with schema.org types when available. * curl * Go ``` curl "https://tourism.api.opendatahub.com/v1/Hotel/123" \ -H "Accept: application/ld+json" \ -H "Authorization: Bearer $TOKEN" ``` ``` req, _ := http.NewRequest("GET", "https://tourism.api.opendatahub.com/v1/Hotel/123", nil) req.Header.Set("Accept", "application/ld+json") req.Header.Set("Authorization", "Bearer "+token) ``` warning JSON-LD format is not supported for all data types. Currently, the Content API supports JSON-LD for the following types: * Hotel * Restaurant * Event * TouristAttraction * Recipe * SkiResort * Place For unsupported types, the API returns standard JSON regardless of the `Accept` header. ## 2. Response structure and metadata[​](#2-response-structure-and-metadata "Direct link to 2. Response structure and metadata") All Content API responses include metadata that describes the result set, along with the actual data in the `Items` array. ### 2.1. Search and list response metadata[​](#21-search-and-list-response-metadata "Direct link to 2.1. Search and list response metadata") When using endpoints like `/v1/Filter` or `/v1/Find`, the response includes pagination and summary metadata: | Field | Type | Description | | -------------- | ------- | ------------------------------------------ | | `TotalResults` | integer | Total number of results matching the query | | `TotalPages` | integer | Total number of pages available | | `CurrentPage` | integer | Current page number in the result set | | `Seed` | string | Random seed used for consistent pagination | | `Items` | array | Array of result objects | Example response: ``` { "TotalResults": 10564, "TotalPages": 705, "CurrentPage": 1, "OnlineResults": -1, "Seed": "43", "Items": [ { "Id": "AF3EA703C8563631497597C0ADAF75B6", "AccoDetail.en.Name": "Appartement Drei Zinnen/Three Peaks", "_Meta.Type": "accommodation", "Self": "https://tourism.api.opendatahub.com/v1/Accommodation/AF3EA703C8563631497597C0ADAF75B6" } ] } ``` The `Seed` parameter ensures consistent sorting across paginated requests. If you retrieve the first page with `Seed: "43"`, use that same seed value when requesting subsequent pages to avoid duplicate or missing results. ### 2.2. Individual result fields[​](#22-individual-result-fields "Direct link to 2.2. Individual result fields") Each item in the `Items` array contains core fields: | Field | Description | | ---------------------------------------- | ---------------------------------------- | | `Id` | Unique identifier for the resource | | `Type` or `_Meta.Type` | Content type of the resource | | `Title` or language-specific title field | Display name of the resource | | `Link` or `Self` | Direct URL to retrieve the full resource | ## 3. CSV export[​](#3-csv-export "Direct link to 3. CSV export") The Content API supports CSV output for flat data structures. You can request CSV format either via query parameter or HTTP header. ### 3.1. Requesting CSV format[​](#31-requesting-csv-format "Direct link to 3.1. Requesting CSV format") Use one of these methods to receive CSV output: * Add `format=csv` to the query string * Set the `Accept` header to `text/csv` - curl - Go ``` curl "https://tourism.api.opendatahub.com/v1/Accommodation?fields=Id,AccoDetail.en.Name,AccoType.Id&format=csv" \ -H "Authorization: Bearer $TOKEN" ``` ``` req, _ := http.NewRequest("GET", "https://tourism.api.opendatahub.com/v1/Accommodation", nil) q := req.URL.Query() q.Add("fields", "Id,AccoDetail.en.Name,AccoType.Id") q.Add("format", "csv") req.URL.RawQuery = q.Encode() req.Header.Set("Authorization", "Bearer "+token) ``` ### 3.2. Field selection and flattening[​](#32-field-selection-and-flattening "Direct link to 3.2. Field selection and flattening") Use the `fields` parameter to specify which fields to include in the CSV output. This helps create flat, tabular data suitable for spreadsheet applications. Example request: ``` https://tourism.api.opendatahub.com/v1/Accommodation?fields=Id,AccoDetail.en.Name,AccoType.Id&format=csv ``` warning CSV export requires flat data structures. Nested objects may not export correctly or may be omitted. For complex data, use JSON format and transform locally if needed. | Parameter | Type | Required | Default | Description | | --------- | ------ | -------- | ---------------- | --------------------------------------------------------- | | `format` | string | no | json | Output format; use `csv` for CSV export | | `fields` | string | no | all fields | Comma-separated list of fields to include in the response | | `Accept` | header | no | application/json | Set to `text/csv` to request CSV format | warning Not all endpoints support CSV export. If the `format=csv` parameter or `Accept: text/csv` header is not honored, the endpoint only supports JSON output. ## 4. Data licensing[​](#4-data-licensing "Direct link to 4. Data licensing") All responses from the Content API are provided under an Open Data license. However, resources linked within the data (such as images, websites, or documents) may have different licensing terms. Always verify the license of linked external resources before reuse. The Open Data license applies only to the structured data returned by the API, not to third-party content referenced within it. Related: [Authentication](/use-data/authentication-and-access/authentication.md), Query parameters (Content API) --- # Content API reference The Content API provides access to tourism-related datasets such as accommodations, activities, events, weather, and more. It exposes a set of standardized endpoints for retrieving structured data, with consistent query parameters, response formats, and metadata across all entity types. All endpoints follow a common pattern and return data in a uniform JSON envelope. This reference documents the shared conventions, available entity types, filtering capabilities, and utility endpoints. ## 1. Base URL and entry points[​](#1-base-url-and-entry-points "Direct link to 1. Base URL and entry points") The base URL for the Content API is: ``` https://tourism.api.opendatahub.com/v1/ ``` You can browse available endpoints directly at this URL. For interactive exploration, use the Swagger UI: warning A JSON browser plugin may be required to properly view raw API responses in your browser. ## 2. Entity endpoints[​](#2-entity-endpoints "Direct link to 2. Entity endpoints") The Content API exposes multiple entity types, each accessible via a dedicated endpoint. The general pattern is: * `GET /v1/{EntityType}` — returns a paged list of items * `GET /v1/{EntityType}/{id}` — returns a single item by ID All list responses use the same JSON envelope structure: ``` { "TotalResults": 150, "TotalPages": 15, "CurrentPage": 1, "Seed": null, "Items": [ // Array of entity objects ] } ``` ### 2.1. Main entity types[​](#21-main-entity-types "Direct link to 2.1. Main entity types") | Entity Type | Endpoint | Description | | -------------------------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------ | | `Accommodation` | `/v1/Accommodation` | Lodging facilities including hotels, guesthouses, and agriturismos | | `ODHActivityPoi` | `/v1/ODHActivityPoi` | Activities, points of interest, and gastronomy venues | | `Event` / `EventShort` | `/v1/Event`, `/v1/EventShort` | Cultural, sports, and seasonal events | | `Article` | `/v1/Article` | Editorial content and stories | | `Webcam` | `/v1/WebcamInfo` | Webcam streams and metadata | | `Weather` | `/v1/Weather/District`, `/v1/Weather/Forecast`, `/v1/Weather/Realtime` | Weather data by district and forecast | | `Region`, `Municipality`, `District`, `Area` | `/v1/Region`, etc. | Administrative and tourism regions | | `SkiArea`, `SkiRegion` | `/v1/SkiArea`, `/v1/SkiRegion` | Ski resort areas | | `TourismAssociation` | `/v1/TourismAssociation` | Local tourism organizations | ## 3. Shared query parameters[​](#3-shared-query-parameters "Direct link to 3. Shared query parameters") The following parameters are supported across most Content API endpoints. ### 3.1. Pagination[​](#31-pagination "Direct link to 3.1. Pagination") | Parameter | Type | Required | Default | Description | | ------------ | ------- | -------- | ------- | ----------------------------------- | | `pagesize` | integer | no | 10 | Number of items per page (max 1000) | | `pagenumber` | integer | no | 1 | Page number to retrieve | * curl ``` curl "https://tourism.api.opendatahub.com/v1/Accommodation?pagesize=10&pagenumber=2" ``` ### 3.2. Field selection[​](#32-field-selection "Direct link to 3.2. Field selection") | Parameter | Type | Required | Default | Description | | --------- | ------ | -------- | ------- | ----------------------------------------------------- | | `fields` | string | no | - | Comma-separated list of fields to include in response | Supports nested fields and array indexing: * `fields=Shortname` — include top-level field * `fields=Detail.en.Title` — include localized title * `fields=ODHTags.[*].Id` — return array of tag IDs - curl ``` curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?fields=Id,Shortname,Detail.en.Title" ``` ### 3.3. Language handling[​](#33-language-handling "Direct link to 3.3. Language handling") | Parameter | Type | Required | Default | Description | | ------------ | ------ | -------- | ------- | ------------------------------------------------------------------ | | `language` | string | no | - | Return only content in specified language (e.g., `en`, `de`, `it`) | | `langfilter` | string | no | - | Filter results to those available in the specified language | * curl ``` curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?language=de&langfilter=de" ``` ### 3.4. Search and filtering[​](#34-search-and-filtering "Direct link to 3.4. Search and filtering") | Parameter | Type | Required | Default | Description | | ------------------ | ----------------- | -------- | ------- | ---------------------------------------------------------------------------------- | | `searchfilter` | string | no | - | Full-text search on title fields across languages; accepts partial matches and IDs | | `removenullvalues` | boolean | no | false | Exclude null fields from response to reduce payload size | | `updatefrom` | date (yyyy-MM-dd) | no | - | Return only items updated on or after the given date | * curl ``` curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?searchfilter=hiking&removenullvalues=true&updatefrom=2023-01-01" ``` ### 3.5. Geospatial filtering[​](#35-geospatial-filtering "Direct link to 3.5. Geospatial filtering") | Parameter | Type | Required | Default | Description | | ----------------------- | ------- | -------- | ------- | -------------------------------------------------------------------------------------------- | | `latitude`, `longitude` | float | no | - | Coordinates for radius-based filtering | | `radius` | integer | no | - | Radius in meters; returns items within the circle defined by latitude, longitude, and radius | | `polygon` | string | no | - | Filter by geographic polygon using WKT, custom syntax, or GeoShapes ID | | `areafilter` | string | no | - | Filter by area name (e.g., ski area); does not support wildcards | * curl ``` # By radius curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?latitude=46.624975&longitude=11.369909&radius=2000" # By polygon (GeoShapes ID) curl "https://tourism.api.opendatahub.com/v1/WebcamInfo?polygon=8032_swisstopo" # By polygon (WKT) curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?polygon=POLYGON((11.026805%2046.688285,11.083110%2046.690169,11.081394%2046.660723,11.035045%2046.655775,11.026805%2046.688285))" ``` info The `polygon` parameter supports: * `Country.Type.Id` or `Country.Type.Name` (e.g., `it.municipality.Bolzano/Bozen`) * WKT format (POLYGON, LINESTRING, MULTIPOLYGON) * BBC/BBI syntax * Custom SRID via `;SRID=4326` suffix Large polygons may exceed URL length limits. ### 3.6. Tag-based filtering[​](#36-tag-based-filtering "Direct link to 3.6. Tag-based filtering") | Parameter | Type | Required | Default | Description | | ----------- | ------ | -------- | ------- | ------------------------------------------------------------------------------------------------------------ | | `tagfilter` | string | no | - | Filter by tag identifiers. Comma-separated values use OR logic; use `and(...)`/`or(...)` for explicit logic. | Tag filtering is supported on the `ODHActivityPoi` endpoint. Tag values are case-sensitive lowercase identifiers (for example `hiking`), not display labels; discover the valid tags for an entity with the [`/v1/Tag`](#41-tags) endpoint. An unknown tag returns zero results rather than an error. * curl ``` # OR logic (any of the listed tags) curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?tagfilter=hiking,bike" # AND logic (all of the listed tags) curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi?tagfilter=and(hiking,summer)" ``` info Two tagging systems are used: * `LTSTags`: original tags from data sources * `ODHTags`: Open Data Hub standardized categories Use `ODHTags` for consistent categorization. The `AdditionalPoiInfos.[language].Categories` field contains display-ready categories per language. ## 4. Utility endpoints[​](#4-utility-endpoints "Direct link to 4. Utility endpoints") ### 4.1. Tags[​](#41-tags "Direct link to 4.1. Tags") Retrieve tags used for categorization. * **Current endpoint**: `GET /v1/Tag` * **Deprecated**: `/v1/ODHTag` (do not use) Supports filtering by: * `validforentity` (e.g., `odhactivitypoi`, `event`, `accommodation`) * `source` (e.g., `lts`, `idm`) * `types` (tag category) - curl ``` curl "https://tourism.api.opendatahub.com/v1/Tag?validforentity=odhactivitypoi&source=lts" ``` ### 4.2. Distinct values[​](#42-distinct-values "Direct link to 4.2. Distinct values") Retrieve unique values for specific fields. * `GET /v1/Distinct?type=tag&fields=Types.[*]&getasarray=true` — get all tag types * `GET /v1/Distinct?odhtype=odhmetadata&fields=Type&getasarray=true` — get all entity types - curl ``` curl "https://tourism.api.opendatahub.com/v1/Distinct?type=tag&fields=Types.[*]&getasarray=true" ``` ### 4.3. GeoShapes[​](#43-geoshapes "Direct link to 4.3. GeoShapes") Access geographic boundaries for regions, municipalities, and other administrative units. * `GET /v1/GeoShapes` — returns geometry data in GeoJSON format * Supports `srid` parameter to transform coordinate systems (default: EPSG:4326) Used in conjunction with the `polygon` filter parameter. * curl ``` curl "https://tourism.api.opendatahub.com/v1/GeoShapes?polygon=it.region.Trentino-Alto%20Adige" ``` ### 4.4. GeoConverter[​](#44-geoconverter "Direct link to 4.4. GeoConverter") Convert KML or GPX files to GeoJSON. * `GET /v1/GeoConverter/KmlToGeoJson?url={url}` * `POST /v1/GeoConverter/KmlToGeoJson` with file upload * Similar endpoints for GPX conversion - curl ``` curl "https://tourism.api.opendatahub.com/v1/GeoConverter/KmlToGeoJson?url=https://example.com/data.kml" ``` ### 4.5. Location[​](#45-location "Direct link to 4.5. Location") Retrieve location metadata for use with `locfilter`. * `GET /v1/Location?showall=true` — get all location types and IDs Used to obtain `typ` and `id` values for location-based filtering. * curl ``` curl "https://tourism.api.opendatahub.com/v1/Location?showall=true" ``` ### 4.6. Search endpoints[​](#46-search-endpoints "Direct link to 4.6. Search endpoints") * `GET /v1/Find` — search across multiple entity types * `GET /v1/Filter` — advanced filtering with expression language - curl ``` curl "https://tourism.api.opendatahub.com/v1/Find?q=hiking" ``` ### 4.7. Source[​](#47-source "Direct link to 4.7. Source") Retrieve valid data sources. * `GET /v1/Source` — list all source identifiers The `Source` field in data records now contains a single value derived from this endpoint. * curl ``` curl "https://tourism.api.opendatahub.com/v1/Source" ``` ## 5. Response format and metadata[​](#5-response-format-and-metadata "Direct link to 5. Response format and metadata") ### 5.1. Content negotiation[​](#51-content-negotiation "Direct link to 5.1. Content negotiation") The Content API supports multiple response formats via the `Accept` header: * `application/json` (default) * `text/csv` * `application/ld+json` (JSON-LD with schema.org context) ### 5.2. \_Meta section[​](#52-_meta-section "Direct link to 5.2. _Meta section") Every entity includes a `_Meta` object with system metadata: | Field | Description | | ------------------ | --------------------------------------------------- | | `_Meta.Id` | Unique record identifier | | `_Meta.Source` | Data origin (e.g., `lts`, `idm`) | | `_Meta.LastUpdate` | Timestamp when data was last saved by Open Data Hub | | `_Meta.LastChange` | Timestamp of last actual data modification | | `_Meta.Reduced` | Indicates whether data has been reduced | info `_Meta.LastUpdate` reflects system import time, not data change time. Use `_Meta.LastChange` to detect actual content updates. ## 6. Authentication[​](#6-authentication "Direct link to 6. Authentication") Some datasets require authentication to access full details. * Include a valid access token in the `Authorization` header: ``` Authorization: Bearer ``` * Tokens can be obtained through the Swagger UI or OAuth2 flow. warning Accommodation room data and availability require authorized access. Without a token, only base data is returned. ## 7. Example: Filtering ODHActivityPoi[​](#7-example-filtering-odhactivitypoi "Direct link to 7. Example: Filtering ODHActivityPoi") The `ODHActivityPoi` endpoint supports rich filtering. This example combines multiple parameters: * curl ``` curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi\ ?tagfilter=hiking\ &language=en\ &fields=Id,Shortname,Detail.en.Title,Tags\ &pagesize=10" ``` This request: * Filters for hiking-related POIs * Returns only English content * Selects specific fields * Pages results ## 8. Related[​](#8-related "Direct link to 8. Related") * [Authentication guide](/use-data/authentication-and-access/authentication.md) * [Deprecations](/use-data/content-api/deprecations.md) * [Filtering and sorting](/use-data/content-api/filtering-and-sorting.md) --- # Content API explorer Try the Content API directly in your browser below. You can also open the full [Swagger UI on the API host](https://tourism.api.opendatahub.com/swagger/index.html), which is the most reliable place to run live requests. Loading Swagger UI... --- # Geo API The Geo API provides fast, map-optimized access to Open Data Hub geospatial datasets — points of interest, hiking trails, cycleways, and more — served as vector tiles for efficient rendering at any zoom level. It is designed specifically to solve performance and quota issues that come from loading large geometry-heavy datasets through the Content (REST) API. Use the Geo API to **display data on a map**; use the [Content API](/use-data/content-api/reference.md) to **fetch full detail records**. ## 1. Base URL and entry points[​](#1-base-url-and-entry-points "Direct link to 1. Base URL and entry points") The base URL for the Geo API is: ``` https://geo.api.opendatahub.com/ ``` For interactive exploration, use the Swagger UI: Worked examples showing how to display Geo API data on a map: Source code: ## 2. When to use the Geo API vs. the Content API[​](#2-when-to-use-the-geo-api-vs-the-content-api "Direct link to 2. When to use the Geo API vs. the Content API") | Use case | Recommended API | | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | Displaying points or geometries on a map | **Geo API** | | Displaying the full detail of a single item | **Content (REST) API** | | Weather forecast data | Content (REST) API only | | People Counter / time series data | Content (REST) API only (Timeseries support on the Geo API is planned — see [Known limitations](#5-known-limitations)) | ## 3. Operation modes[​](#3-operation-modes "Direct link to 3. Operation modes") The Geo API supports three `operationmode` values, which control how geometries are returned: * `points` — returns point geometries only * `tracks` — returns line/track geometries only * `pointsandtracks` — returns both By default, tracks become visible starting at **zoom level 12**. This can be overridden with the `displaytracksonzoomlevel` parameter. ## 4. Datasets[​](#4-datasets "Direct link to 4. Datasets") ### 4.1. SpatialData endpoint (recommended)[​](#41-spatialdata-endpoint-recommended "Direct link to 4.1. SpatialData endpoint (recommended)") For datasets such as HikingTrails, Cycleways, and similar geometry-based POIs, use the new `SpatialData` endpoint. It replaces the older `ODHActivityPoi` / `GeoShape` endpoints, which are now considered obsolete for this purpose. The `Announcement` endpoint is unaffected by this change and continues to work as before. ### 4.2. Large datasets[​](#42-large-datasets "Direct link to 4.2. Large datasets") Two datasets available through `SpatialData` are very large (roughly 600,000 records each): * `euregio.roadnetwork` * `euregio.routes` These can still be retrieved, but response times may be slow. Caching for these sources is planned but not yet implemented (work in progress). ## 5. Known limitations[​](#5-known-limitations "Direct link to 5. Known limitations") * **Weather forecasts** are served directly by the province and cannot currently be processed into vector tiles, so they remain available only through the Content (REST) API. * **People Counter** data is only available via the Timeseries REST API. Timeseries support on the Geo API is planned for a future release. * **Caching** for the large `euregio.roadnetwork` and `euregio.routes` sources has not yet been implemented, so expect slower responses for these two sources specifically. ## 6. Getting started[​](#6-getting-started "Direct link to 6. Getting started") 1. Browse available layers and try requests directly in [Swagger](https://geo.api.opendatahub.com/swagger). 2. Review the [worked examples](https://geo.api.opendatahub.com/examples/) for map integration patterns (e.g. MapLibre/Leaflet vector tile layers). 3. Choose an `operationmode` (`points`, `tracks`, or `pointsandtracks`) based on what you need to display. 4. For anything requiring full record detail rather than map display, switch to the corresponding Content API endpoint (e.g. `ODHActivityPoi/{id}`, `Announcement/{id}`). ## 7. Related[​](#7-related "Direct link to 7. Related") * [Content API reference](https://docs.opendatahub.com/use-data/content-api/reference) * [Geo API GitHub repository](https://github.com/noi-techpark/opendatahub-geo-api) * [Geo API Swagger UI](https://geo.api.opendatahub.com/swagger) * [Geo API examples](https://geo.api.opendatahub.com/examples/) --- # Filtering time series data The Time Series API supports flexible filtering of mobility data using the `where` query parameter with dot-notation operators. This enables precise retrieval of station and time series data based on attribute values, spatial relationships, and logical combinations. Unlike SQL, the filter syntax uses a functional, dot-separated format that is both human-readable and URL-safe. ## 1. Filter syntax and operators[​](#1-filter-syntax-and-operators "Direct link to 1. Filter syntax and operators") Filtering in the Time Series API is performed via the `where` parameter, which accepts expressions using dot-notation operators. These operators compare field values against literals or expressions and return matching records. ### 1.1 Supported operators[​](#11-supported-operators "Direct link to 1.1 Supported operators") The following operators are available for constructing filter expressions: | Operator | Description | Example | | ---------- | ---------------------------- | ------------------------------------------------- | | `eq` | Equal to | `scode.eq."P123"` | | `neq` | Not equal to | `sorigin.neq."internal"` | | `lt` | Less than | `mvalue.lt.50` | | `gt` | Greater than | `mvalue.gt.100` | | `lteq` | Less than or equal to | `smetadata.capacity.lteq.200` | | `gteq` | Greater than or equal to | `smetadata.capacity.gteq.50` | | `re` | Regular expression match | `sname.re."Parking.*"` | | `ire` | Case-insensitive regex match | `sname.ire."parking.*"` | | `in` | Value is in list | `scode.in.("P1","P2","P3")` | | `nin` | Value is not in list | `scode.nin.("P4","P5")` | | `bbi` | Bounding box intersects | `scoordinate.bbi.(11.1,46.1,11.4,46.5,4326)` | | `bbc` | Bounding box contains | `scoordinate.bbc.(11.1,46.1,11.4,46.5,4326)` | | `dlt` | Distance less than (meters) | `scoordinate.dlt.(3000,11.158682,46.668853,4326)` | | `and(...)` | Logical AND of conditions | `and(scode.eq."P1",mvalue.gt.50)` | | `or(...)` | Logical OR of conditions | `or(scode.eq."P1",scode.eq."P2")` | info Rules verified against the live API: * String values must be wrapped in double quotes: `scode.eq."105"`. An unquoted string returns HTTP 400. Numeric values take no quotes: `mvalue.gt.50`. In a shell, wrap the whole URL in single quotes so the double quotes are sent literally. * Geospatial and list operators use the form `field.operator.(args)` with a dot before the parenthesis (`scoordinate.bbi.(...)`, `scode.in.("P1","P2")`). Omitting the dot returns a parse error. * Station metadata fields are addressed under `smetadata` (for example `smetadata.capacity`), not as bare names. Measurement fields such as `mvalue` are only available when the request targets a data endpoint (a `{dataTypes}` segment, for example `/v2/flat/ParkingStation/*`). warning The `tagfilter` and `source` parameters are not supported in the Time Series API. These are specific to the Content API and will be ignored if used in Time Series API requests. ### 1.2 Logical operators[​](#12-logical-operators "Direct link to 1.2 Logical operators") You can combine multiple conditions using `and(...)` and `or(...)` functions. These accept comma-separated filter expressions as arguments. For example, to retrieve measurements above 50 for the parking station with code `P123` (a `{dataTypes}` segment, here `*`, is required to filter on the measurement field `mvalue`): * Go * curl * Python * JavaScript ``` url := `https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation/*?where=and(scode.eq."P123",mvalue.gt.50)` resp, err := http.Get(url) ``` ``` curl 'https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation/*?where=and(scode.eq."P123",mvalue.gt.50)' ``` ``` import requests url = "https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation/*" params = {"where": 'and(scode.eq."P123",mvalue.gt.50)'} response = requests.get(url, params=params) ``` ``` fetch('https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation/*?where=and(scode.eq."P123",mvalue.gt.50)') .then(response => response.json()) .then(data => console.log(data)); ``` ## 2. Spatial filtering[​](#2-spatial-filtering "Direct link to 2. Spatial filtering") The Time Series API provides several operators for geographic filtering of station data. ### 2.1 Distance-based filtering with `dlt`[​](#21-distance-based-filtering-with-dlt "Direct link to 21-distance-based-filtering-with-dlt") The `dlt` (distance less than) operator filters stations within a specified radius (in meters) from a geographic point. It uses the WGS84 coordinate system (SRID 4326) by default. Syntax: `coordinate.dlt.(distance, longitude, latitude, SRID?)` (note the dot before the parenthesis) * `distance`: radius in meters (required) * `longitude`: point-x (required) * `latitude`: point-y (required) * `SRID`: spatial reference system (optional, defaults to 4326) - Go - curl - Python - JavaScript ``` resp, err := http.Get("https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation?where=scoordinate.dlt.(3000,11.158682,46.668853,4326)") ``` ``` curl "https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation?where=scoordinate.dlt.(3000,11.158682,46.668853,4326)" -H "Authorization: Bearer $TOKEN" ``` ``` import requests url = "https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation" params = {"where": "scoordinate.dlt.(3000,11.158682,46.668853,4326)"} headers = {"Authorization": "Bearer "} response = requests.get(url, params=params, headers=headers) ``` ``` fetch('https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation?where=scoordinate.dlt.(3000,11.158682,46.668853,4326)', { headers: { 'Authorization': 'Bearer ' } }) ``` warning Ensure the coordinate order is (longitude, latitude). Many mapping tools display coordinates as (latitude, longitude), which will produce incorrect results if used directly. ### 2.2 Bounding box filtering[​](#22-bounding-box-filtering "Direct link to 2.2 Bounding box filtering") For rectangular area filtering, use `bbi` (bounding box intersects) or `bbc` (bounding box contains). * `bbi`: returns stations whose coordinates intersect the box * `bbc`: returns stations whose coordinates are fully contained in the box Syntax: `coordinate.bbi.(minx,miny,maxx,maxy,SRID?)` or `coordinate.bbc.(...)` (note the dot before the parenthesis) Example: retrieve stations within bounding box around Bolzano ``` https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation?where=scoordinate.bbi.(11.1,46.1,11.4,46.5,4326) ``` ## 3. Field selection and pagination[​](#3-field-selection-and-pagination "Direct link to 3. Field selection and pagination") In addition to filtering, you can control the response format and volume using other query parameters. ### 3.1 Selecting fields[​](#31-selecting-fields "Direct link to 3.1 Selecting fields") Use the `select` parameter to specify which fields to include in the response. This reduces payload size and improves performance. Example: retrieve only station code and capacity ``` https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation?select=scode,smetadata.capacity&where=smetadata.capacity.gteq.100 ``` ### 3.2 Pagination[​](#32-pagination "Direct link to 3.2 Pagination") Use `limit` and `offset` to paginate large result sets. * `limit`: number of records per page (default varies by endpoint) * `offset`: number of records to skip Example: get second page of 50 records ``` https://mobility.api.opendatahub.com/v2/flat,node/ParkingStation?limit=50&offset=50 ``` Other supported parameters include `distinct`, `shownull`, `timezone`, and `origin`. ## 4. Best practices and caveats[​](#4-best-practices-and-caveats "Direct link to 4. Best practices and caveats") warning Avoid overly restrictive filters that may return empty results. Complex `and(...)` conditions especially can exclude all data if any condition is unmet. * Test filters incrementally, starting with broad conditions and narrowing down * Use `or(...)` when you need alternative conditions; `and(...)` requires all conditions to match * Validate coordinate order when using spatial filters * Be aware that field names (like `scoordinate`) may vary by station type; consult the API documentation for your specific dataset * The `dlt` operator provides a more intuitive alternative to bounding box filters for proximity searches For more information on available station types and data models, see the [Time Series API reference](/use-data/time-series-api/reference.md). --- # Time Series API reference The Time Series API provides access to mobility-related time series data through a unified endpoint structure. This reference describes the API's base structure, available representations, filtering capabilities, query parameters, and response format. ## 1. Base structure and endpoints[​](#1-base-structure-and-endpoints "Direct link to 1. Base structure and endpoints") The Time Series API uses a single, consistent endpoint pattern to access all mobility datasets. The base host is `https://mobility.api.opendatahub.com`. ### 1.1. Endpoint pattern[​](#11-endpoint-pattern "Direct link to 1.1. Endpoint pattern") The general path structure is: ``` /v2/{representation}/{stationTypes}/{dataTypes} ``` Optionally followed by: * `/latest` to retrieve the most recent data point * `/{from}/{to}` to retrieve data within a time range (half-open interval: `from` is included, `to` is excluded) The `representation` parameter combines two aspects: * Structure: `flat` or `tree` * Type: `node`, `edge`, or `event` (optional, comma-separated) For example: `flat,node`, `tree,edge` ### 1.2. Available endpoints[​](#12-available-endpoints "Direct link to 1.2. Available endpoints") | Endpoint | Description | | ----------------------------------------------------------------- | ----------------------------------------------------------------- | | `GET /v2/{representation}` | Returns the list of available representations | | `GET /v2/{representation}/{stationTypes}` | Returns metadata for stations of the given type(s) | | `GET /v2/{representation}/{stationTypes}/{dataTypes}` | Returns time series data for the specified station and data types | | `GET /v2/{representation}/{stationTypes}/{dataTypes}/latest` | Returns the latest data point | | `GET /v2/{representation}/{stationTypes}/{dataTypes}/{from}/{to}` | Returns data within the specified time range | Use `*` in the `{dataTypes}` (or `{stationTypes}`) segment to match all values. For example, `GET /v2/flat/ParkingStation/*/latest` returns the latest measurement for every data type of every parking station. The `{from}` and `{to}` bounds accept `yyyy-MM-dd` or a full `yyyy-MM-ddTHH:mm:ss` timestamp. ### 1.3. Example: Retrieve all ParkingStation data[​](#13-example-retrieve-all-parkingstation-data "Direct link to 1.3. Example: Retrieve all ParkingStation data") * Go * curl * Python * JavaScript ``` package main import ( "fmt" "net/http" "io" ) func main() { url := "https://mobility.api.opendatahub.com/v2/flat/ParkingStation" resp, err := http.Get(url) if err != nil { panic(err) } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Println(string(body)) } ``` ``` curl "https://mobility.api.opendatahub.com/v2/flat/ParkingStation" ``` ``` import requests url = "https://mobility.api.opendatahub.com/v2/flat/ParkingStation" response = requests.get(url) print(response.json()) ``` ``` fetch('https://mobility.api.opendatahub.com/v2/flat/ParkingStation') .then(response => response.json()) .then(data => console.log(data)); ``` ## 2. Representations[​](#2-representations "Direct link to 2. Representations") The Time Series API supports different data representations to suit various use cases. ### 2.1. Structure: flat vs tree[​](#21-structure-flat-vs-tree "Direct link to 2.1. Structure: flat vs tree") * `flat`: All metadata and data fields are at the same level. Suitable for simple consumption and flat data models. * `tree`: Preserves hierarchical relationships between data elements. Useful when parent-child relationships matter. ### 2.2. Type: node, edge, event[​](#22-type-node-edge-event "Direct link to 2.2. Type: node, edge, event") * `node`: Represents a measurement station (e.g., a parking sensor or charging station). Includes all associated metadata. * `edge`: Represents a connection between two stations (e.g., a road segment). Includes start station, end station, and edge-specific metadata (prefixed with `e`, such as `eactive`). * `event`: Represents discrete events in the mobility network. The type can be combined with the structure using a comma, for example: `flat,node` or `tree,edge`. info The `node` representation is equivalent to the pre-2020.10 API output. For backward compatibility, the type can be omitted in requests. ## 3. Query parameters[​](#3-query-parameters "Direct link to 3. Query parameters") The Time Series API supports several query parameters to control the response. | Parameter | Type | Required | Default | Description | | ---------- | ------- | -------- | ------- | ------------------------------------------------------------ | | `where` | string | no | - | Filter expression using DOT-NOTATION operators | | `select` | string | no | - | Comma-separated list of fields to include in the response | | `limit` | integer | no | 200 | Maximum number of results to return. Use -1 to disable limit | | `offset` | integer | no | 0 | Number of results to skip (for pagination) | | `distinct` | boolean | no | true | Whether to return distinct results | | `shownull` | boolean | no | - | Whether to include null values in the response | | `timezone` | string | no | - | Timezone for datetime fields | | `origin` | string | no | - | Filter by data origin | Pagination is handled using `limit` and `offset`. For example, to retrieve results 200 to 399, use `limit=200&offset=200`. ## 4. Filtering with the where parameter[​](#4-filtering-with-the-where-parameter "Direct link to 4. Filtering with the where parameter") The `where` parameter supports expressive filtering using DOT-NOTATION operators. ### 4.1. Comparison operators[​](#41-comparison-operators "Direct link to 4.1. Comparison operators") | Operator | Meaning | | -------- | ----------------------------------------- | | `eq` | Equal to | | `neq` | Not equal to | | `lt` | Less than | | `gt` | Greater than | | `lteq` | Less than or equal to | | `gteq` | Greater than or equal to | | `re` | Regular expression match | | `ire` | Case-insensitive regular expression match | | `in` | In a list of values | | `nin` | Not in a list of values | ### 4.2. Geospatial operators[​](#42-geospatial-operators "Direct link to 4.2. Geospatial operators") | Operator | Meaning | | -------- | ----------------------------------------------------------------- | | `bbi` | Bounding box intersection: returns items partially within the box | | `bbc` | Bounding box containment: returns items completely within the box | Coordinates must be specified in **longitude, latitude** order. Geospatial operators use the form `field.operator.(args)` with a dot before the parenthesis, for example `scoordinate.bbi.(11.1,46.1,11.4,46.5,4326)`. Omitting that dot returns an HTTP 400 parse error. ### 4.3. Logical operators[​](#43-logical-operators "Direct link to 4.3. Logical operators") * `and(...)` : Logical AND between conditions * `or(...)` : Logical OR between conditions ### 4.4. Examples[​](#44-examples "Direct link to 4.4. Examples") warning String values in a `where` expression must be wrapped in double quotes, for example `scode.eq."105"`. An unquoted string value returns an HTTP 400 error. Numeric values are written without quotes (`mvalue.gt.50`). In a shell, wrap the whole URL in single quotes so the double quotes are sent literally. #### Filter by station code[​](#filter-by-station-code "Direct link to Filter by station code") * curl * Go ``` curl 'https://mobility.api.opendatahub.com/v2/flat/ParkingStation?where=scode.eq."105"' ``` ``` url := `https://mobility.api.opendatahub.com/v2/flat/ParkingStation?where=scode.eq."105"` ``` #### Filter measurements and combine conditions[​](#filter-measurements-and-combine-conditions "Direct link to Filter measurements and combine conditions") To filter on a measurement field such as `mvalue`, request a data endpoint by adding a `{dataTypes}` segment (`*` matches all data types). Retrieve measurements above 50 for the station with code `105`: ``` curl 'https://mobility.api.opendatahub.com/v2/flat/ParkingStation/*?where=and(scode.eq."105",mvalue.gt.50)' ``` #### Filter by bounding box[​](#filter-by-bounding-box "Direct link to Filter by bounding box") Geospatial operators use the form `field.operator.(args)` (note the dot before the parenthesis). Retrieve stations within a geographic area (longitude 11.1 to 11.4, latitude 46.1 to 46.5, SRID 4326): ``` curl "https://mobility.api.opendatahub.com/v2/flat/ParkingStation?where=scoordinate.bbi.(11.1,46.1,11.4,46.5,4326)" ``` warning Always use longitude, latitude order. Using latitude, longitude will result in incorrect filtering. ## 5. Response format[​](#5-response-format "Direct link to 5. Response format") All Time Series API responses are JSON objects with a consistent envelope structure: ``` { "offset": 0, "limit": 200, "data": [ // Array of result objects ] } ``` * `offset`: The number of skipped results * `limit`: The maximum number of results returned * `data`: The actual payload, an array of objects containing the requested data When `limit` is set to `-1`, all results are returned without pagination. ## 6. OpenAPI specification[​](#6-openapi-specification "Direct link to 6. OpenAPI specification") The complete API is documented in the OpenAPI specification, available at: You can explore and test the API interactively using the Swagger UI at: ## 7. Example station types[​](#7-example-station-types "Direct link to 7. Example station types") The Time Series API supports multiple station types. Common examples include: * `ParkingStation`: Parking facilities and sensors * `EChargingStation`: Electric vehicle charging stations * `Flight`: Flight-related mobility data These are not separate APIs but different values for the `{stationTypes}` path parameter. ## 8. Related[​](#8-related "Direct link to 8. Related") * [Quickstart: your first request](/quickstart.md) * [Filtering time series data](/use-data/time-series-api/filtering.md) * [Authentication for protected datasets](/use-data/authentication-and-access/authentication.md) --- # Time Series API explorer Try the Time Series API directly in your browser below. You can also open the full [Swagger UI](https://swagger.opendatahub.com/?url=https://mobility.api.opendatahub.com/v2/apispec), which is the most reliable place to run live requests. Loading Swagger UI... --- # Transmodel API: NeTEx and SIRI-Lite The Transmodel API provides standardized access to both static and real-time mobility data in compliance with European and Italian interoperability standards. It supports two key formats: NeTEx for scheduled data and SIRI-Lite for real-time updates. This API is part of the MaaS4Italy initiative and integrates with the Open Data Hub, enabling third-party applications to retrieve mobility information through the National Access Point (NAP). ## 1. NeTEx Endpoints[​](#1-netex-endpoints "Direct link to 1. NeTEx Endpoints") The Transmodel API delivers static mobility data in NeTEx (Network Timetable Exchange) format, the European standard for public transport schedules, infrastructure, and related services. These endpoints are designed for applications requiring consistent, planned data such as timetables, parking facilities, and shared mobility services. The Transmodel API ensures compatibility with the NeTEx Italian profile, supporting national interoperability requirements under the MaaS4Italy initiative. ### 1.1. Available Endpoints[​](#11-available-endpoints "Direct link to 1.1. Available Endpoints") All NeTEx endpoints use the `GET` method and return data in XML format according to the NeTEx schema. | Endpoint | Description | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `/netex` | Returns a full NeTEx export of all available data from the Open Data Hub, including public transport schedules and infrastructure. | | `/netex/parking` | Provides a partial NeTEx export focused on parking facilities, including locations, capacities, and metadata. | | `/netex/flights` | Delivers flight-related data in NeTEx format. | | `/netex/sharing` | Includes data on bike- or car-sharing services, such as station locations, available vehicles, and operator information. | warning The NeTEx format is intended for static, scheduled data and may not be suitable for real-time use cases. Parsing NeTEx XML responses requires familiarity with the schema structure and may involve additional processing on the client side. ### 1.2. Example Request[​](#12-example-request "Direct link to 1.2. Example Request") Retrieve the full NeTEx dataset: * Go * curl * Python * JavaScript ``` package main import ( "fmt" "net/http" ) func main() { url := "https://transmodel.api.opendatahub.com/netex" resp, err := http.Get(url) if err != nil { panic(err) } defer resp.Body.Close() fmt.Printf("Status: %s\n", resp.Status) // Handle XML response } ``` ``` curl "https://transmodel.api.opendatahub.com/netex" ``` ``` import requests url = "https://transmodel.api.opendatahub.com/netex" response = requests.get(url) print(f"Status: {response.status_code}") print(response.text) # XML content ``` ``` fetch('https://transmodel.api.opendatahub.com/netex') .then(response => { console.log(`Status: ${response.status}`); return response.text(); }) .then(data => console.log(data)) // XML content .catch(error => console.error('Error:', error)); ``` ## 2. SIRI-Lite Endpoints[​](#2-siri-lite-endpoints "Direct link to 2. SIRI-Lite Endpoints") The Transmodel API includes SIRI-Lite (Service Interface for Real-Time Information) endpoints to deliver real-time mobility data. These RESTful APIs provide live updates on parking availability and shared mobility services, enabling dynamic applications such as journey planners and mobility dashboards. The base URL for all SIRI-Lite services is `https://efa.sta.bz.it`. ### 2.1. Facility Monitoring Endpoints[​](#21-facility-monitoring-endpoints "Direct link to 2.1. Facility Monitoring Endpoints") These endpoints support real-time monitoring of parking and sharing facilities. | Endpoint | Description | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `/siri-lite/facility-monitoring` | General endpoint for real-time updates on both parking and sharing facilities, including status, occupancy, and availability. | | `/siri-lite/facility-monitoring/parking` | Provides real-time data on parking facility status, available spaces, and occupancy. | | `/siri-lite/facility-monitoring/sharing` | Delivers real-time data on shared mobility services, including available vehicles, operator details, and station status. | ### 2.2. Other SIRI-Lite Services[​](#22-other-siri-lite-services "Direct link to 2.2. Other SIRI-Lite Services") Additional SIRI-Lite services are published by STA and accessible under the same base URL. | Endpoint | Description | | -------------------------------- | ------------------------------------------------------------------------------------------------- | | `/siri-lite/estimated-timetable` | Exchanges estimated schedules in real time, including updates to planned departures and arrivals. | | `/siri-lite/vehicle-monitoring` | Provides information about vehicle movement and progress against the scheduled timetable. | | `/siri-lite/situation-exchange` | Delivers information about incidents, disruptions, or planned events affecting public transport. | ### 2.3. Filtering Options[​](#23-filtering-options "Direct link to 2.3. Filtering Options") The SIRI-Lite endpoints support flexible query parameters to refine results. | Filter | Description | | ------------- | -------------------------------------------------------------------------------------------------------------------- | | `facilityRef` | Filter by a specific facility identifier (e.g., parking lot or sharing station ID). | | `location` | Retrieve data for facilities near a geographic point, specified as latitude and longitude (e.g., `46.4983,11.3548`). | | `operator` | Narrow results by the operator of the service (e.g., bike-sharing provider). | ### 2.4. Example Request[​](#24-example-request "Direct link to 2.4. Example Request") Fetch real-time status for all parking facilities: * Go * curl * Python * JavaScript ``` package main import ( "fmt" "net/http" ) func main() { url := "https://efa.sta.bz.it/siri-lite/facility-monitoring/parking" resp, err := http.Get(url) if err != nil { panic(err) } defer resp.Body.Close() fmt.Printf("Status: %s\n", resp.Status) // Handle XML or JSON response } ``` ``` curl "https://efa.sta.bz.it/siri-lite/facility-monitoring/parking" ``` ``` import requests url = "https://efa.sta.bz.it/siri-lite/facility-monitoring/parking" response = requests.get(url) print(f"Status: {response.status_code}") print(response.text) ``` ``` fetch('https://efa.sta.bz.it/siri-lite/facility-monitoring/parking') .then(response => { console.log(`Status: ${response.status}`); return response.text(); }) .then(data => console.log(data)) .catch(error => console.error('Error:', error)); ``` warning Authentication requirements and rate limits for the SIRI-Lite endpoints are not publicly documented. If you encounter access issues or need higher quotas, contact for support. ## 3. Additional Resources[​](#3-additional-resources "Direct link to 3. Additional Resources") * **Swagger Documentation**: Explore the Transmodel API interactively using the [Swagger interface](https://swagger.opendatahub.com/?url=https://transmodel.api.opendatahub.com/apispec#/). * **GitHub Repository**: Access source code, validation scripts, and tools in the [Transmodel API GitHub repository](https://github.com/noi-techpark/transmodel-api). * **National Access Point**: The Transmodel API is integrated with the Open Data Hub and accessible through the National Access Point (NAP), serving as the official gateway for MaaS4Italy data distribution. Visit the [NAP catalog](https://www.cciss.it/nap/mmtis/public/catalog/Organisation/1381) for published datasets. For further assistance, contact the Open Data Hub team at . --- # Transmodel API explorer Try the Transmodel API directly in your browser below. You can also open the full [Swagger UI](https://swagger.opendatahub.com/?url=https://transmodel.api.opendatahub.com/apispec), which is the most reliable place to run live requests. Loading Swagger UI... --- # Domains and datasets The Open Data Hub provides structured access to datasets across multiple domains through REST APIs. As an API user or ingestion creator, you interact with these datasets via standardized endpoints, metadata, and response formats. This guide covers the core concepts you need to navigate and retrieve data effectively. ## 1. Data domains[​](#1-data-domains "Direct link to 1. Data domains") Datasets are grouped into **domains**. **Mobility** and **Tourism** are the main focus right now, but other domains are covered as well: * **Mobility**: public transport, parking, e-charging stations, and other station and sensor data. * **Tourism**: accommodations, events, points of interest, and activities. * **Other**: additional datasets that do not fall under the two main domains. Each dataset is accessible through a REST API endpoint, described in its metadata (see the next section). ## 2. The Metadata API[​](#2-the-metadata-api "Direct link to 2. The Metadata API") The Open Data Hub publishes a catalog of every dataset through the **Metadata API**. It is the source the [Discovery tool](/tools/discovery.md) is built on, and you can query it directly: * `GET https://tourism.api.opendatahub.com/v1/MetaData` returns the full catalog of datasets. * `GET https://tourism.api.opendatahub.com/v1/MetaData/{id}` returns a single dataset's metadata. Each record describes one dataset and tells you how to consume it. The most useful fields are: | Field | Description | | ---------------------- | ------------------------------------------------------------------------------------ | | `Shortname` | Human-readable name of the dataset. | | `BaseUrl` / `ApiUrl` | The API host and the endpoint URL where the dataset is served. | | `ApiType` | Which API exposes the dataset (for example, content or timeseries). | | `OdhType` / `Category` | The dataset's type and thematic categories. | | `ApiAccess` | Whether the dataset is `open` or `closed` (closed datasets require authentication). | | `RecordCount` | Record totals, for example `RecordCount.Total`. | | `SwaggerUrl` | Link to the Swagger UI for the dataset's endpoint. | | `LicenseInfo` | The license under which the data is published (see [Data licensing](/licensing.md)). | | `Deprecated` | `true` if the dataset should no longer be used. | info Use the Metadata API, or the [Discovery tool](/tools/discovery.md) built on top of it, to find a dataset's endpoint, access level, and license before you start querying it. ## 3. API structure and endpoints[​](#3-api-structure-and-endpoints "Direct link to 3. API structure and endpoints") APIs in the Open Data Hub follow consistent patterns within each domain, but the exact structure varies by API. Always refer to the Swagger documentation for the specific domain you are working with. ### 3.1. Content API[​](#31-content-api "Direct link to 3.1. Content API") The Content API, which serves the Tourism domain, uses the following endpoint patterns: * `GET /v1/{Name}`: Returns the full dataset. * `GET /v1/{Name}/{Id}`: Returns a specific item by its ID. Here, `{Name}` is the case-sensitive `ApiIdentifier` of the dataset (e.g., `ODHActivityPoi`), and `{Id}` is the unique identifier of a data object. * Go * curl * Python * JavaScript ``` // Retrieve all accommodations resp, err := http.Get("https://tourism.api.opendatahub.com/v1/Accommodation") ``` ``` curl "https://tourism.api.opendatahub.com/v1/Accommodation" ``` ``` import requests response = requests.get("https://tourism.api.opendatahub.com/v1/Accommodation") ``` ``` fetch('https://tourism.api.opendatahub.com/v1/Accommodation') .then(response => response.json()) .then(data => console.log(data)); ``` ### 3.2. Time Series API[​](#32-time-series-api "Direct link to 3.2. Time Series API") The Time Series API, which serves the Mobility domain, returns paginated responses with the following structure: ``` { "offset": 0, "limit": 100, "data": [ // Array of station or measurement objects ] } ``` * `offset`: Number of items to skip (default: 0). * `limit`: Maximum number of items to return (default: 200; use -1 to disable). * `data`: The actual payload of the response. This structure supports pagination. To simulate page-by-page results, increment the `offset` value by the `limit` in successive requests. info The `limit` parameter defaults to 200. Setting it to -1 disables the limit and returns all results, which may impact performance. ## 4. Dataset type and filtering[​](#4-dataset-type-and-filtering "Direct link to 4. Dataset type and filtering") The `Type` field in the `_Meta` section identifies the category of a data object and is used across APIs for filtering and validation. * The `Type` is a string (e.g., `accommodation`, `eventshort`) and is unique per dataset. * When using the Search API or Distinct Search API, you must provide a valid `type` parameter. * Invalid or unregistered types result in an API error. For example, to search for all accommodations: ``` GET /v1/Search?type=accommodation ``` warning The `type` parameter must match a valid Dataset Type exactly. Custom types are not allowed without registration. ## 5. Access control and dataset availability[​](#5-access-control-and-dataset-availability "Direct link to 5. Access control and dataset availability") Dataset access is determined by the `ApiAccess` field: * `open`: The dataset is publicly accessible. * `closed`: The dataset requires authentication and appropriate role assignment. Only open datasets include the `RecordCount` field in their metadata. For closed datasets, this field is omitted due to access restrictions. To read a closed dataset, see [Authentication and authorization](/use-data/authentication-and-access/authentication.md). ## 6. Related[​](#6-related "Direct link to 6. Related") * [Quickstart: your first request](/quickstart.md) * [Content API reference](/use-data/content-api/reference.md) and [Filtering and sorting](/use-data/content-api/filtering-and-sorting.md) * [Time Series API reference](/use-data/time-series-api/reference.md) * [Authentication and authorization](/use-data/authentication-and-access/authentication.md) --- # Data licensing The Open Data Hub provides open, freely accessible data under permissive licenses, ensuring compliance with open data principles. As an API user or ingestion creator, you can rely on structured license metadata to understand the terms under which data can be used. ## 1. License information in API responses[​](#1-license-information-in-api-responses "Direct link to 1. License information in API responses") The Content API includes a `LicenseInfo` block in every dataset response. This block provides key metadata about the licensing of the data record and is present for all publicly exposed data. ### 1.1. Structure of `LicenseInfo`[​](#11-structure-of-licenseinfo "Direct link to 11-structure-of-licenseinfo") The `LicenseInfo` object contains the following fields: | Field | Type | Description | | --------------- | ------- | ------------------------------------------------------------------ | | `Author` | string | The author of the data. May be empty. | | `License` | string | The license type applied to the data (e.g., `CC0`, `Proprietary`). | | `ClosedData` | boolean | Indicates whether the data is closed (`true`) or open (`false`). | | `LicenseHolder` | string | URL or identifier of the entity holding the license. | info The `LicenseInfo` block applies **only** to the parent data node (the one identified by an `Id`). It does not extend to child nodes such as `Streamurl` or `Webcamurl`, which may link to external resources governed by different licensing terms. ### 1.2. Common license values[​](#12-common-license-values "Direct link to 1.2. Common license values") * **`CC0`**: The data is in the public domain and can be freely reused without restrictions. * **`Proprietary`**: The data is owned by a specific entity and may have usage limitations. Only open data records are returned through the Content API. Even if source systems contain closed or proprietary data, these are filtered out during ingestion and not exposed in API responses. ### 1.3. Example `LicenseInfo` in a response[​](#13-example-licenseinfo-in-a-response "Direct link to 13-example-licenseinfo-in-a-response") ``` { "Id": "D3659E1F111C4CDB2EC19F8FC95118B7", "Active": true, "Webcamurl": "https://webtv.feratel.com/webtv/?&pg=5EB12424-7C2D-428A-BEFF-0C9140CD772F", "LicenseInfo": { "Author": "", "License": "CC0", "ClosedData": false, "LicenseHolder": "https://www.lts.it" } } ``` In this example, the `LicenseInfo` applies only to the parent record (e.g., the webcam metadata), not to the stream or image content accessible via `Webcamurl`. ## 2. Image licensing and availability[​](#2-image-licensing-and-availability "Direct link to 2. Image licensing and availability") Images associated with data records are subject to additional licensing checks. The Content API only includes images that are licensed under permissive terms. ### 2.1. Image license requirements[​](#21-image-license-requirements "Direct link to 2.1. Image license requirements") * Images must have a `CC0` or `CC-BY` license to be included in API responses. * Images with no license (`License == null`) or a proprietary license (e.g., `LTS`) are excluded. This filtering ensures compliance with open data standards. As a result, not all accommodations or points of interest include images in their responses. ### 2.2. Per-image license details[​](#22-per-image-license-details "Direct link to 2.2. Per-image license details") When images are included, license information is provided in the `ImageGallery` object: | Field | Type | Description | | --------------- | ------ | -------------------------------- | | `ImageUrl` | string | URL of the image. | | `License` | string | License type (e.g., `CC0`). | | `CopyRight` | string | Copyright notice, if applicable. | | `LicenseHolder` | string | Entity holding the license. | Example: ``` { "ImageUrl": "https://urltoanimage", "License": "CC0", "CopyRight": "© LTS", "LicenseHolder": "https://www.lts.it" } ``` ## 3. Access and usage considerations[​](#3-access-and-usage-considerations "Direct link to 3. Access and usage considerations") ### 3.1. No authentication required[​](#31-no-authentication-required "Direct link to 3.1. No authentication required") Open Data Hub data can be accessed without authentication. You can make requests directly to the API endpoints. warning To avoid CORS errors in browser-based applications, always use `https://opendatahub.com` (not HTTP) when calling the API from JavaScript. ### 3.2. Company details as open data[​](#32-company-details-as-open-data "Direct link to 3.2. Company details as open data") Company-related information such as names, addresses, and contact details are treated as open data and can be freely shared with third parties. These do not contain personal data and are published under open licensing terms. ## 4. Time series data licensing[​](#4-time-series-data-licensing "Direct link to 4. Time series data licensing") Licensing for the Time Series API applies at two levels: * **Station data** (the station catalog: identifiers, names, coordinates, and metadata) is generally published under **CC0** for all stations. * **Measurements** are generally **CC0 when the dataset is open**. Measurements that belong to a **closed** dataset are **proprietary** and require authentication and the appropriate role to read. To check whether a dataset is open or closed, see its `ApiAccess` value in the [Metadata API](/domains-and-datasets.md#2-the-metadata-api) or the [Discovery tool](/tools/discovery.md). ## Related[​](#related "Direct link to Related") * [Filtering and sorting (Content API)](/use-data/content-api/filtering-and-sorting.md) * [Output formats (Content API)](/use-data/content-api/output-formats.md) * [Quickstart: your first request](/quickstart.md) --- # Quickstart: your first request This guide follows the path most developers take: find the dataset you need, inspect it, then read it from the API. Most Open Data Hub data is open and needs no authentication. ## 1. Discover the dataset[​](#1-discover-the-dataset "Direct link to 1. Discover the dataset") Start at [Discovery](https://discovery.opendatahub.com) to find the right dataset and understand its structure. You can search datasets by keyword, tag, or domain, inspect each dataset's schema (its fields and their types), and read field-level statistics that tell you whether a field is actually populated. See the [Discovery tool page](/tools/discovery.md) for details. Knowing the dataset and the fields you need up front saves you from guessing at parameters later. ## 2. Inspect the data[​](#2-inspect-the-data "Direct link to 2. Inspect the data") Before (or instead of) writing code, you can look at the data directly: * **[Data Browser](/tools/data-browser/.md)**: browse and filter tourism content (events, accommodations, points of interest) in a table and see the exact fields the Content API returns. * **[Analytics](/tools/analytics.md)**: chart mobility time series over a chosen time range to see how a measurement behaves before you query it. ## 3. Make your first request[​](#3-make-your-first-request "Direct link to 3. Make your first request") Every dataset is a REST endpoint that returns JSON. ### 3.1. Content API[​](#31-content-api "Direct link to 3.1. Content API") To retrieve activity points of interest (POIs), send a `GET` request to the `ODHActivityPoi` endpoint: * Go * curl * Python * JavaScript ``` package main import ( "fmt" "io" "net/http" ) func main() { resp, err := http.Get("https://tourism.api.opendatahub.com/v1/ODHActivityPoi") if err != nil { panic(err) } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Println(string(body)) } ``` ``` curl "https://tourism.api.opendatahub.com/v1/ODHActivityPoi" ``` ``` import requests response = requests.get("https://tourism.api.opendatahub.com/v1/ODHActivityPoi") print(response.json()) ``` ``` fetch('https://tourism.api.opendatahub.com/v1/ODHActivityPoi') .then(response => response.json()) .then(data => console.log(data)); ``` info The Content API returns a response envelope with `TotalResults`, `TotalPages`, `CurrentPage`, and an `Items` array. For the full list of endpoints, parameters, and filters, see the [Content API reference](/use-data/content-api/reference.md). ### 3.2. Time Series API[​](#32-time-series-api "Direct link to 3.2. Time Series API") To retrieve all parking stations, call the `flat` representation of the `ParkingStation` type: * Go * curl * Python * JavaScript ``` package main import ( "fmt" "io" "net/http" ) func main() { resp, err := http.Get("https://mobility.api.opendatahub.com/v2/flat/ParkingStation") if err != nil { panic(err) } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) fmt.Println(string(body)) } ``` ``` curl "https://mobility.api.opendatahub.com/v2/flat/ParkingStation" ``` ``` import requests response = requests.get("https://mobility.api.opendatahub.com/v2/flat/ParkingStation") print(response.json()) ``` ``` fetch('https://mobility.api.opendatahub.com/v2/flat/ParkingStation') .then(response => response.json()) .then(data => console.log(data)); ``` info The Time Series API returns an envelope with `offset`, `limit`, and a `data` array, and caps results at 200 by default. Page through more with `limit` and `offset`. For the full endpoint structure and the `where` filter, see the [Time Series API reference](/use-data/time-series-api/reference.md). ### 3.3. Try it in Swagger[​](#33-try-it-in-swagger "Direct link to 3.3. Try it in Swagger") Both APIs publish an interactive Swagger UI where you can build and run requests in the browser: * [Content API Swagger](https://tourism.api.opendatahub.com/swagger/index.html) * [Time Series API Swagger](https://swagger.opendatahub.com/?url=https://mobility.api.opendatahub.com/v2/apispec) ## 4. Next steps[​](#4-next-steps "Direct link to 4. Next steps") * Read the [Content API reference](/use-data/content-api/reference.md) for tourism entities, parameters, and filtering. * Read the [Time Series API reference](/use-data/time-series-api/reference.md) for the mobility endpoint structure and the `where` filter. * Learn about [authentication](/use-data/authentication-and-access/authentication.md) if you need access to restricted data. * Explore [quotas, the CLI, and the R package](/use-data/authentication-and-access/quotas-and-tools.md) for alternative access methods. ---