7 Data Collection

This chapter summarises how data was collected, validated, harmonised, and processed within Open Music Europe. It connects the practical work of WP1–WP4 to the data-to-policy pipeline described in Section 2.5 and to the governing principles defined in the Data Management Plan. It also introduces the software components developed in WP4, which implement the project’s methodological requirements and prepare datasets for integration into the Open Music Observatory.

Because the DMP has not yet been updated to reflect all datasets collected during implementation, this chapter includes placeholders indicating where the DMP manager must add or revise content. Once the DMP is updated, this chapter will be synchronised with it and will become the operational reference for data ingestion.

After the DMP is updated, all data-collection workflows and summaries presented here will be aligned with the approved Data Summaries and metadata structures.

Not updated

Because of the serious delay in WP6 with the updating of the Data Management Plan, this section should not be reviewed until harmonised with the DMP.

7.1 Overview of the Data-Collection Framework

Data collection in Open Music Europe followed the pipeline logic defined in Section 2.5. Each thematic work package identified indicators and conceptual models (D1.1, D2.1, D3.1), which determined the data inputs required for analysis. These sources were then evaluated based on accessibility, legal compliance, interoperability, and their relevance to indicator construction.

The data-collection effort covered four categories:

administrative and register data
survey data
statistical and economic data
platform and streaming data

The workflows for collecting, accessing, and harmonising these sources were originally documented in internal working files and the first version of the DMP. They must now be consolidated into the updated DMP.

The DMP manager must add all final sources, access conditions, and descriptions to the Data Summaries section for traceability.

7.2 Administrative and Register Data

Administrative datasets provided essential inputs to WP1 and WP3, where economic valuation, labour structures, and participation indicators required high-resolution administrative signals. These workflows included:

accessing royalty and licensing records from CMOs
collecting grant and programme data from ministries
extracting public business and organisation registers
harmonising venue, festival, and event registers
obtaining public-sector microdata in the Slovak pilot, including historical KULT survey data

These datasets formed sampling frames, validation structures, and reference datasets for linking surveys and platform data to legal and organisational entities.

The DMP must describe each administrative source, its controller, its legal basis, and its access pathway.

7.3 Survey Data

Survey data in WP2 and WP3 filled gaps that administrative and platform data could not address. Survey work included:

enterprise surveys of MSMEs in the music sector
personal surveys on participation, wellbeing, and music behaviour
experimental modules on diversity, mobility, and cultural citizenship
harmonisation with national statistical surveys, especially the Slovak KULT survey

Survey workflows required careful attention to:

sampling-frame construction
questionnaire metadata
informed-consent procedures
pseudonymisation and controlled processing
harmonisation using SDMX, DDI, and GSIM concepts

The DMP must contain questionnaire versions, variable lists, consent protocols, and pseudonymisation details for each survey.

7.4 Statistical and Economic Data

Statistical sources enabled European and national comparability across indicators. These included:

national accounts and satellite cultural accounts
labour-force data
business demography and structural-business statistics
external-trade and export data
cultural consumption and household budget surveys
price indices and cost-structure information

These datasets were central to the economic modelling in WP1, circulation and diversity indicators in WP2, and societal-impact work in WP3.

The DMP must include references, licences, and access conditions for all statistical datasets used.

7.5 Platform and Streaming Data

Platform datasets (WP1, WP2, WP4) were collected via API-based sampling and automated scripts, including:

Spotify API samples (popularity, metadata, audio features)
YouTube API samples
playlist-localisation datasets
automated crawlers for repertoire discovery

These data supported: - digital-market structure analysis - validation of economic indicators - repertoire and rights linkage - circulation and localisation studies

Licensing and terms-of-service constraints mean these datasets have strict reuse limitations.

The DMP must describe permitted uses and restrictions for each platform dataset.

7.6 Processing and Harmonisation

After collection, datasets underwent several processing stages aligned with the architecture described in Chapter 4.

These stages included:

cleaning and transformation
pseudonymisation of personal data
cross-linking with authority files and identifiers
metadata enrichment
structural harmonisation (SDMX, DataCite, DDI)
conversion to formats suitable for ingestion into the OMO

The openmusic-pipeline (WP4) implemented these processes in R, ensuring reproducibility and alignment with FAIR and OPA principles.

The updated DMP must list controlled vocabularies, classifications, and harmonisation rules used.

7.7 Software Components Developed in WP4

WP4 created a suite of open-source tools that implement the pipeline and prepare data for the OMO. These are described here briefly, with technical detail deferred to annexes.

7.7.1 Data-Ingestion Tools

These tools handle imports from Excel, CSV, SQL exports, APIs, and legacy systems. They include: - connectors for CMO and/or ministry datasets - survey-import scripts - API wrappers for streaming platforms

7.7.2 Validation and Reconciliation Tools

These tools perform semantic alignment and quality checks: - authority-control reconciliation (ISNI, VIAF, ORCID, corporate registries) - SPARQL-based constraint checks - duplicate detection and entity merging tools

7.7.3 Harmonisation and Metadata Tools

These implement the project’s semantic rules:

SDMX structure builders
DDI variable metadata generators
DataCite dataset metadata templates
vocabulary management utilities

7.7.4 OMO Integration Tools

These tools prepare processed data for the Observatory:

Wikibase ingestion scripts
URI stabilisation and PID mapping utilities
JSON-LD and RDF exporters
R and Python client libraries for the OMO API

The DMP must reference the software components used for metadata production and processing, as required by Horizon Europe guidelines.

7.8 Position of Data Collection and Processing in the Pipeline

Data collection and processing bridge the conceptual work of WP1–WP3 and the semantic and technical infrastructure described in Chapter 4. They provide:

validated raw inputs
harmonised metadata
cross-domain entity linking
analysis-ready datasets

These steps ensure the construction of traceable, reproducible indicators and policy outputs.

7.9 Integration with the Open Music Observatory

Once the DMP is updated and the Data Summaries approved, all data collections and processing workflows described here will be integrated into the OMO:

datasets will be linked to persistent identifiers
metadata will be converted to human- and machine-readable forms
provenance will be documented in accordance with OPA and FAIR
ingestion routines will run automatically using WP4 tools

Ingestion will begin once the DMP contains the complete, updated Data Summaries and metadata structures for all datasets.