Refactor catalog for handling disjoint intervals of data

[faysou]

Pull Request

• Refactor catalog for handling disjoint intervals of data

  • Use closed time intervals to represent data contained in a catalog for a given data type
  • Information about time range contained in a parquet file name
  • Saving new data only by creating a new file
  • Possibility to consolidate all files into a single file, or only a given range of time into a single file for maintance purpose
  • Possiblity to reset file names of an existing catalog to work with the new format by reading the min and max timestamps of a parquet file and changing the file name
  • Taking into account empty intervals into file names when possible in order to create contiguous files (timestamps are integers so the notion of contiguity exists there), in order to avoid repeated queries where there's no data for example during weekends
  • Use timestamp information in files to use the minimum amount of parquet files required for a given catalog query

• Improve data engine to handle several market data queries and one catalog query

  • Also ensure that received data from a client or catalog matches the bounds intended in a request
  • Data concatenated from several requests is then sorted
  • The code now assumes that a given symbol is only in one catalog at a time, this simplifies implementation
  • When data is present in a catalog, only the minimum amount of data is queried to a live market data client, which is economical for paid historical data

• Improve handling of custom data in data requests and data subscriptions by better taking into account optional instrument_id

• Migrate catalog implementation to rust as well

• Libraries used for intervals:

  • In python: https://github.com/AlexandreDecan/portion
  • In rust: https://github.com/jonathanGB/Unbounded-Interval-Tree

Type of change

• New feature (non-breaking change which adds functionality)

How has this change been tested?

Existing tests are passing
Added tests for the new catalog features