Module RimuIO

Module to provide file input and output functionality for Rimu. Provides convenience functions:

• RimuIO.save_df(filename, df::DataFrame) Save dataframe in Arrow format.
• RimuIO.load_df(filename) Load Arrow file into dataframe.
• RimuIO.save_dvec(filename, dv) Save dict vector in BSON format.
• RimuIO.load_dvec(filename) Load BSON file into dict vector.

RimuIO.load_df(filename; propagate_metadata = true, add_filename = true) -> DataFrame

Load Arrow file into DataFrame. Optionally propagate metadata to DataFrame and add the file name as metadata.

See also RimuIO.save_df.

RimuIO.load_dvec(filename) -> AbstractDVec

Load AbstractDVec stored in BSON.

RimuIO.save_df(filename, df::DataFrame; kwargs...)

Save dataframe in Arrow format.

Keyword arguments are passed on to Arrow.write. Compression is enabled by default for large DataFrames (over 10,000 rows).

Table-level metadata of the DataFrame is saved as Arrow metadata (with String value) unless overwritten with the keyword argument metadata.

See also RimuIO.load_df.

RimuIO.save_dvec(filename, dvec)

Save dvec in BSON format.

Notes

• Only the localpart is saved. You may need to re-wrap the result in MPIData if using MPI.
• When using this function with MPI, make sure to save the vectors from different ranks to different files, e.g. by saving as RimuIO.save_dvec("filename-$(mpi_rank()).bson", dvec).