Module RimuIO

Module to provide file input and output functionality for Rimu. Provides convenience functions:

• RimuIO.save_df(filename, df::DataFrame) Save dataframe in Arrow format.
• RimuIO.load_df(filename) Load Arrow file into dataframe.
• RimuIO.save_state(filename, vector; metadata...) Save a vector and optinal metadata in Arrow format.
• RimuIO.load_state(filename) Load a file created by save_state.

struct DVecAsTable

Wrapper over the storage of a DVec that allows us to treat a DVec as a table from Tables.jl. Constructed with Tables.table(::DVec).

struct PDVecAsTable

Wrapper over the storage of a PDVec that allows us to treat a PDVec as a table from Tables.jl. Constructed with Tables.table(::PDVec).

load_df(filename; propagate_metadata = true, add_filename = true) -> DataFrame

Load Arrow file into DataFrame. Optionally propagate metadata to DataFrame and add the file name as metadata.

See also RimuIO.save_df.

load_state(filename; kwargs...) -> PDVec, NamedTuple
load_state(PDVec, filename; kwargs...) -> PDVec, NamedTuple
load_state(DVec, filename; kwargs...) -> DVec, NamedTuple

Load the state saved in the Arrow file filename. kwargs are passed to the constructor of PDVec/DVec. Any metadata stored in the file is be parsed as a number (if possible) and returned alongside the vector in a NamedTuple.

See also save_state.

save_df(filename, df::DataFrame; kwargs...)

Save dataframe in Arrow format.

Keyword arguments are passed on to Arrow.write. Compression is enabled by default for large DataFrames (over 10,000 rows).

Table-level metadata of the DataFrame is saved as Arrow metadata (with String value) unless overwritten with the keyword argument metadata.

See also RimuIO.load_df.

save_state(filename, vector; io, kwargs...)

Save PDVec or DVec vector to an arrow file filename.

io determines the output stream to write progress to. Defaults to stderr when MPI is enabled and devnull otherwise.

All other kwargs are saved as strings to the arrow file and will be parsed back when the state is loaded.

See also load_state.