Reference:
Alpatov A.N., Bogatireva A.A..
Data storage format for analytical systems based on metadata and dependency graphs between CSV and JSON
// Software systems and computational methods. – 2024. – ¹ 2.
– P. 1-14.
Read the article
Abstract: In the modern information society, the volume of data is constantly growing, and its effective processing is becoming key for enterprises. The transmission and storage of this data also plays a critical role. Big data used in analytics systems is most often transmitted in one of two popular formats: CSV for structured data and JSON for unstructured data. However, existing file formats may not be effective or flexible enough for certain data analysis tasks. For example, they may not support complex data structures or provide sufficient control over metadata. Alternatively, analytical tasks may require additional information about the data, such as metadata, data schema, etc. Based on the above, the subject of this study is a data format based on the combined use of CSV and JSON for processing and analyzing large amounts of information. The option of sharing the designated data types for the implementation of a new data format is proposed. For this purpose, designations have been introduced for the data structure, which includes CSV files, JSON files, metadata and a dependency graph. Various types of functions are described, such as aggregating, transforming, filtering, etc. Examples of the application of these functions to data are given. The proposed approach is a technique that can significantly facilitate the processes of information analysis and processing. It is based on a formalized approach that allows you to establish clear rules and procedures for working with data, which contributes to their more efficient processing. Another aspect of the proposed approach is to determine the criteria for choosing the most appropriate data storage format. This criterion is based on the mathematical principles of information theory and entropy. The introduction of a criterion for choosing a data format based on entropy makes it possible to evaluate the information content and compactness of the data. This approach is based on the calculation of entropy for selected formats and weights reflecting the importance of each data value. By comparing entropies, you can determine the required data transmission format. This approach takes into account not only the compactness of the data, but also the context of their use, as well as the possibility of including additional meta-information in the files themselves and supporting data ready for analysis.
Keywords: Apache Parquet, Integration of data formats, Data analysis, Analysis Ready Data, Metadata, Data processing, CSV, JSON, Data storage formats, Big Data
References:
Malcolm, R., Morrison, C., Grandison, T., Thorpe, S., Christie, K., Wallace, A., Ñampbell, A. (2014). Increasing the accessibility to big data systems via a common services api. In 2014 IEEE International Conference on Big Data (Big Data), 883-892.
Wu, T. (2009). System of Teaching quality analyzing and evaluating based on Data Warehouse. Computer Engineering and Design, 30(6), 1545-1547.
Vitagliano, G., Hameed, M., Jiang, L., Reisener, L., Wu, E., Naumann, F. (2023). Pollock: A Data Loading Benchmark. Proceedings of the VLDB Endowment, 16(8), 1870-1882.
Xiaojuan, L., Yu, Z. (2023). A data integration tool for the integrated modeling and analysis for east. Fusion Engineering and Design, 195, 113933.
Lemzin, A. (2023). Streaming Data Processing. Asian Journal of Research in Computer Science, 15(1), 11-21.
Hughes, L. D., Tsuen