conesToolBox package
conesToolBox.conesErrors module
- exception conesToolBox.conesErrors.conesKalmanGainInversionError(value, tolerance)[source]
Bases:
ExceptionException raised when the Kalman gain matrix inversion check exceeds tolerance, indicating a poorly conditioned (S@S.T + R) matrix
conesToolBox.conesFunctions module
- conesToolBox.conesFunctions.cones_count_netcdf_obs(file)[source]
Count the number of observations stored in an observation netCDF file
- Parameters:
file (str) – Path to the observation netCDF file
- Returns:
The number of observations
- Return type:
int
- conesToolBox.conesFunctions.cones_get_netcdf_obs_coord(file, index)[source]
Read the (x, y, z) coordinates of one observation from a netCDF file
- Parameters:
file (str) – Path to the observation netCDF file
index (int) – Index of the observation
- Returns:
The [x, y, z] coordinates, or an error message if index is out of bounds
- Return type:
list
- conesToolBox.conesFunctions.cones_get_netcdf_obs_var(file)[source]
List the observed variable names stored in a netCDF file, excluding coordinates (x, y, z) and standard deviation companions (*_std)
- Parameters:
file (str) – path to the observation netCDF file
- Returns:
The list of observed variable names
- Return type:
list
- conesToolBox.conesFunctions.cones_get_netcdf_var(file, index, var)[source]
Read the value of a given observation variable at a given index
- Parameters:
file (str) – Path to the observation netCDF file
index (int) – Index of the observation
var (str) – Name of the variable to read
- Returns:
The variable value, or an error message on failure
- Raises:
ValueError – if ‘var’ is not a variable of the dataset
- conesToolBox.conesFunctions.cones_live_plot(i, log_file, truth, test_name)[source]
Matplotlib FuncAnimation callback: redraws the live convergence plot of a parameter’s ensemble mean and std read from ‘log_file’
- Parameters:
i – Frame index passed by the FuncAnimation (unused)
log_file (str) – Path to the parameter log file (columns: iter, time, mean, std)
truth (float) – Ground-truth value plotted as a reference line
test_name (str) – Title of the plot
- Returns:
None
- conesToolBox.conesFunctions.cones_open_netcdf_dataset(file, **kwargs)[source]
Open an observation netCDF file, working around files whose variables carry a ‘dtype’ (or other xarray-reserved) key directly in their attrs instead of in their encoding. Some xarray versions refuse to CF-decode such variables and raise “failed to prevent overwriting existing key … in attrs” instead of silently moving the key to encoding.
- conesToolBox.conesFunctions.cones_prior_digest(params, decimals=6)[source]
Short reproducible digest of a realised parameter ensemble. :param params: parameter ensemble, either (n_ensemble, n_parameters) :param decimals: rounding applied before hashing :returns: first 16 hexadecimal characters of the sha256 digest :rtype: str
- conesToolBox.conesFunctions.cones_run_id(conesSet, driver_file)[source]
Unique, self-describing identifier for this run
Three fields joined by underscores: <UTC timestamp> when the run started cfg<8 hex> digest of conesDict, identifies the configuration g<7 jex> short git commit of the sources, or “nogit” :param conesSet: settings object in use :param driver_file: pass __file__ from the calling driver :returns: the run identifier :rtype: str
- conesToolBox.conesFunctions.cones_write_prior_block(path, params, label='forecast', decimals=6)[source]
Append a digest and summary of a parameter ensemble to the run log. :param path: run log to append to; nothing is written if it is None :param params: parameter ensemble as gathered, either orientation :param label: name of the ensemble, for drivers carrying more than one
(fine and coarse for MGEnKF, principal, control and ancillary for MFEnKF)
- Returns:
the digest, or None if nothing was written
- Return type:
str or None
- conesToolBox.conesFunctions.cones_write_run_header(conesSet, driver_file, world_ranks, n_proc_app, run_id=None, path=None)[source]
Truncate path and write the full provenance of this run as comment lines, so that the file identifies itself without external notes.
Call once, before the assimilation loop, from the same rank that appends the per-cycle rows. The per-cycle writes stay in append mode and are unaffected: this call is what makes that append deterministic, instead of concatenating successive runs into one file.
- Parameters:
conesSet – settings object in use (conesSetHF in main_MFEnKF.py)
driver_file – pass __file__ from the calling driver
world_ranks – total number of MPI ranks
n_proc_app – MPI ranks per model instance
path – destination file, default “par_<run_id>.log
- Returns:
the path actually written
- Return type:
str
- conesToolBox.conesFunctions.global2Local(globalID, globalCellList)[source]
Find the rank and local cell index corresponding to a global cell id :param globalID: The globall cell id to locate :type globalID: int :param globalCellList: list of global cell ids per rank :type globalCellList: list
- Returns:
(rank, localID) if found, otherwise None
- Return type:
tuple:
- conesToolBox.conesFunctions.is_cell_set(filepath)[source]
Check the OpenFOAM ‘class’ header of a polyMesh/sets/ file to confirm it is a cellSet, filtering out faceSet/pointSet artifacts (e.g. ‘wrongFaces’ written by checkMesh) that are not DA clippings.
- Parameters:
filepath (str) – path to the set file
- Returns:
True if the file’s FoamFile class is cellSet
- Return type:
bool
- conesToolBox.conesFunctions.local2Global(localID, globalCellList, rank)[source]
Convert a local cell id on a given rank to its global cell id
- Parameters:
localID (int) – the local cell id
globalCellList (list) – List of the global cell ids per rank
rank (int) – The rank the local cell id belongs to
- Returns:
The global cell id
- Return type:
int
- conesToolBox.conesFunctions.matrix_to_txt(matrix, name_of_file)[source]
Dump a matrix to a comma-separated text file for debugging purposes
- Parameters:
matrix (numpy ndarray) – The matrix to dump
name_of_file (str) – Output file name, without extension (“.txt” is appended)
- Returns:
None
- conesToolBox.conesFunctions.mergeClips(clipList, globalCellList)[source]
Given a list of conesClip, finds clippings that shares cells and merge them
- Parameters:
clipList (list) – the list of clippings
globalCellList (list) – the list of global cell ids
- Returns:
the list of global cells id of merged clips
- Return type:
list
conesToolBox.conesMPI module
conesToolBox.conesClasses subpackage
conesClasses.conesClip module
conesClasses.conesObservation module
- class conesToolBox.conesClasses.conesObservation.conesStaticObservation[source]
Bases:
objectA class representing a single static (fixed position) observation, backed by an observation netCDF file
- get_coordinates_from_netcdf()[source]
Read and set the observation’s (x, y, z) coordinates from the netCDF file
- Returns:
None
- get_time_from_netcdf()[source]
Read and set the observation’s time series from the netCDF file
- Returns:
None
- get_vars_from_netcdf(obsVar)[source]
Read the requested observed variables (and their standard deviation) at this observation’s id and time series, and set them as attributes
- Parameters:
obsVar – The list of variable names to read
- Returns:
None
- set_current_time_index(time)[source]
Set the index of this observation’s time series closest to ‘time’
- Parameters:
time (float) – The current simulation time
- Returns:
None
- set_file(file_path)[source]
Set the path to the observation netCDF file
- Parameters:
file_path (str) – The observation netCDF file path”
- Returns:
None
- set_global_cell(cell_obs, obs_ids)[source]
Set the observation’s global cell id on each of its owning ranks
- Parameters:
cell_obs (list) – list, per rank, of global cell ids for the observations held by that rank
obs_ids (list) – list, per rank, of observation ids held by that rank
- Returns:
None
- set_id(id_obs)[source]
Set the observation id, used to index it in the netCDF file
- Parameters:
id_obs (int) – The observation id
- Returns:
None
conesClasses.conesParams module
conesClasses.conesProfiler module
- class conesToolBox.conesClasses.conesProfiler.conesProfiler(n_ensemble, n_decomp, run_id=None)[source]
Bases:
objectA ligthweight profiler tracking wall-clock time spent in the Solver, MPI_Transfer and DA phases of a DA cycle, and the peak resident memory usage, dumped to a csv report.
- reset_cycle()[source]
Zero the per-phase timers for the next cycle. Peak_RAM_GB is intentionally not reset (run-lvel running maximum.
- save_to_csv(filename=None)[source]
Append the accumulated results as one row to a CSV file, writing the header first if the file does not exist yet
- Parameters:
filename (str) – output CSV file path. When omitted, it is derived from the run id, so each run gets its own file.
- Returns:
None
conesClasses.conesRegion module
- class conesToolBox.conesClasses.conesRegion.conesRegion[source]
Bases:
objectA class to define a Data Assimilation region
- addObs(obs)[source]
Method to include an observation to the region
- Parameters:
obs (conesStaticObservation) – The observation
- Returns:
None
- filterSampling(sampling, sampling_order)[source]
From the complete sampling matrix, this function sorts and filters the sampled values for the region
- Parameters:
sampling (numpy ndarray) – The raw sampling matrix gathered
sampling_order (int) – A list of observation id ordered by gather
- Returns:
None
- filterState(state, stateCells)[source]
From the complete state matrix, this function filters the values belonging to the region
- Parameters:
state (numpy ndarray) – The raw state matrix gathered
stateCells (list) – the list of cells id of the state
- Returns:
None
- get_distance_matrix()[source]
Compute the euclidean distance matrix between the region’s state cells and its observations, used for state covaraince localisation
- Returns:
Distance matrix, shaped (n_state_cells, n_observations)
- Return type:
numpy ndarray
- setInflation(conesSettings)[source]
Set the hyperparameters related to Inflation :param conesSettings: the cones object containing the settings of the calculation :type conesSettings: conesSettings class :return: None
- setObsSettings(conesSettings)[source]
Copy the observation-operator settings from the cones settings object
- setParams(params)[source]
Set the parameters associated to the region
- Parameters:
params (numpy ndarray) – array of parameters
- Returns:
None
- setState(state)[source]
Set the state vector for the region
- Parameters:
state (numpy ndarray) – State Matrix
- Returns:
None
- set_cells_coordinates(mesh_dir)[source]
Read the region’s cell center coordinates from an OpenFOAM ‘C’ field, used for state covariance localisation :param mesh_dir: Path to the OpenFOAM case directory :type mesh_dir: str :raises FileNotFoundError: if the cell-centres file was not generated :returns: None
conesClasses.conesEnKF module
- class conesToolBox.conesClasses.conesEnKF.conesEnKF(region)[source]
Bases:
objectA class containing all the basic elements to perform the Ensemble Kalman filter analysis phase
- anomaly(x)[source]
Calculate an anomaly matrix
- Parameters:
x (numpy ndarray) – Matrix containing the ensemble members data
- Returns:
The anomaly matrix
- Return type:
numpy ndarray
- calculateInflation(matrix, coefficient)[source]
Applies stochastic inflation on a matrix
- Parameters:
arr (numpy ndarray) – the matrix to be inflated
infl (float) – inflation coefficient
- Returns:
arr
- Return type:
numpy ndarray
- dump_error_convergence()[source]
Dumps in file “err.log” the norm of the difference between observation and model sampling
- Returns:
None
- dump_params_convergence()[source]
Dumps in file “par.log” the updated parameter ensemble mean and standard deviation
- Returns:
None
- setL(loc_length)[source]
Sets the state covariance localisation matrix :param: loc_length localisation caracteristic length
- setObs(mda_iterations=1, nens=None)[source]
Draws a fresh perturbed observation vector for the currend MDA iteration, using the mean/std already read by set_obs_stats() :params nens: ensemble size to draw the perturbed observations for. Defaults to self.nens :param mda_iterations: inflation coefficient for this iteration :type mda_iterations: int
- Returns:
y
- Return type:
numpy ndarray
- setParams(params)[source]
Set the EnKF parameters
- Parameters:
params (numpy ndarray) – The parameters vector
- Returns:
None
- setSampling(sampling)[source]
Set the EnKF sampling H(x)
- Parameters:
sampling (numpy ndarray) – The sampling matrix
- Returns:
None
- setState(state)[source]
Set the EnKF state
- Parameters:
state (numpy ndarray) – The state vector
- Returns:
None
- set_R(mda_iterations=1)[source]
inflates the observation covariance matrix for the currend MDA iteration, reusing the std computed once by set_obs_stats().
- Parameters:
mda_iterations (int) – inflation coefficient for this iteration
- Returns:
None
- set_obs_stats()[source]
Read, once, the mean and standard deviation of every non-std observation variable in the region at the current time index, caching them for reuse across MDA iterations by set_R() and setObs()
- Returns:
None
- set_x_anomaly()[source]
Recomputes the state-parameters anomaly matrix X. Must be called at every MDA iteration since x is refreshed by update_background(). :returns: None
conesClasses.conesMGRegion module
- class conesToolBox.conesClasses.conesMGRegion.conesMGRegion[source]
Bases:
conesRegionExtends conesRegion with a fine-simulation state for Multi-Grid EnKF. Adds stateFine and filterStateFine to handle the projected state coming from the fine simulation member.
- filterSamplingFine(sampling_proj, sampling_order)[source]
From the global fine-projected sampling matrix, extract and reorder the values belonging to this region’s observations
- Parameters:
sampling_proj (numpy ndarray) – The raw fine-projected sampling column gathered
sampling_order (list) – A list of observation id ordered by gather
- Returns:
None
- filterStateFine(state_proj, globalStateCells)[source]
From the global projected-fine state matrix, extract the rows belonging to this region’s cells. :param state_proj: list of 1-D arrays (one per state variable), each of length
n_coarse_cells (total cells across all coarse procs).
- Parameters:
globalStateCells (list of list of int) – nested list [proc][cell_id] of global cell ids for coarse mesh
conesClasses.conesMGEnKF module
- class conesToolBox.conesClasses.conesMGEnKF.conesMGEnKF(region)[source]
Bases:
conesEnKFMulti-Grid Ensemble Kalman Filter. Extends conesEnKF (coarse ensemble, base class) with the state, parameters and sampling of a single fine-mesh member, updated with the same Kalman gain computed from the coarse ensemvle.
- mergeUpdate()[source]
Concatenate the updated fine member with the updated coarse ensemble into a single upx array :returns: None
conesClasses.conesMFRegion module
- class conesToolBox.conesClasses.conesMFRegion.conesMFRegion[source]
Bases:
conesRegionExtends conesRegion with the two extra data sources needed by Multi-Fidelity EnKF: the control ensemble and the ancillary ensemble, both already projected onto the principal (fine) mesh
- filterSamplingAncillaryProj(sampling_ancillary_proj, sampling_order)[source]
From the global ancillary-projected sampling matrix, extract and reorder the values belonging to this region’s observations
- Parameters:
sampling_ancillary_proj (numpy ndarray) – The raw ancillary-projected sampling matrix gathered
sampling_order (list) – A list of observation id ordered by gather
- Returns:
None
- filterSamplingControlProj(sampling_control_proj, sampling_order)[source]
From the global control-projected sampling matrix, exctract and reorder the values belonging to this region’s observations
- Parameters:
sampling_control_proj (numpy ndarray) – The raw control-projected sampling matrix gathered
sampling_order (list) – A list of observation id ordered by gather
- Returns:
None
- filterStateAncillaryProj(state_ancillary_proj, stateCells)[source]
From the global ancillary-projected state matrix, extract the rows belonging to this region’s cells
- Parameters:
state_ancillary_proj (list of numpy ndarray) – list of 1-D arrays (one per state variable), each spanning all coarse cells
stateCells (list of list of int) – nested list [proc][cell_id] of global cell ids for the coarse mesh
- Returns:
None
- filterStateControlProj(state_control_proj, stateCells)[source]
From the global control-projected state matrix, extract the rows belonging to this region’s cells
- Parameters:
state_control_proj (list of numpy ndarray) – list of 1-D arrays (one per state variable), each spanning all coarse cells
stateCells (list of list of int) – nested list [proc][cell_id] of global cell ids for the coarse mesh
- Returns:
None
- setParamsAncillaryProj(params)[source]
Set the ancillary-projected parameters directly
- Parameters:
params (numpy ndarray) – Ancillary-projected parameters array
- Returns:
None
- setParamsControlProj(params)[source]
Set the control-projected parameters directly
- Parameters:
params (numpy ndarray) – Control-projected parameters array
- Returns:
None
- setStateAncillaryProj(state)[source]
Set the ancillary-projected state directly
- Parameters:
state (numpy ndarray) – Ancillary-projected state array
- Returns:
None
- setStateControlProj(state)[source]
Set the control-projected state directly
- Parameters:
state (numpy ndarray) – Control-projected state array
- Returns:
None
conesClasses.conesMFEnKF module
- class conesToolBox.conesClasses.conesMFEnKF.conesMFEnKF(region)[source]
Bases:
conesEnKFMulti-Fidelity Ensemble Kalman Filter. Extends conesEnKF (principal ensemble, base class) with two extra ensembles already projected onto the principal (fine) mesh: - control: same size as principal, paired 1:1 (en_i <-> control_i) - ancillary: separate, cheaper ensemble, all projected onto en0
- KalmanGainTotal()[source]
Compute the Multi-Fidelity Kalman Gain from the combined covariances of the principal, control and ancillary ensembles
- Returns:
None
- updateParamsAncillaryProj()[source]
Set the updated parameters of the ancillary-projected ensemble
- Returns:
None
- updateParamsControlProj()[source]
Set the updated parameters of the control-projected ensemble
- Returns:
None
- updateStateAncillaryProj()[source]
Set the updated state of the ancillary-projected ensemble
- Returns:
None
conesToolBox.conesClasses.conesSettings subpackage
conesSettings.conesSettingsAbstract module
- class conesToolBox.conesClasses.conesSettings.conesSettingsAbstract.conesSettingsAbstract(case_path, model, nens, n_model_procs)[source]
Bases:
objectA class that groups settings for cones
- checkPath(path)[source]
Check if path exists
- Parameters:
path (str) – The file directory
- Returns:
Boolean
- Return type:
bool
- get_clippings()[source]
Function that reads the clippings in polyMesh/sets/ Returns a list of clippings
- Returns:
list of clippings
- Return type:
list
- mergeClips()[source]
Given a list of conesClip, finds clippings that shares cells and merge them
- Returns:
the list of conesClip after merge
- Return type:
list
- setGlobalCells() list[list[int]][source]
Generate a list of global cell ids splitted by ranks
- Returns:
list of global cells ids
- Return type:
list
- setGlobalStateCells()[source]
Sets a list of global cell ids from which state variables are extracted
- Returns:
None
conesSettings.conesSettings_OF module
- class conesToolBox.conesClasses.conesSettings.conesSettings_OF.conesSettings_OF(case_path, model, nens, n_model_procs)[source]
Bases:
conesSettingsAbstractA class that groups settings for cones
conesSettings.conesSettings_MNH module
- class conesToolBox.conesClasses.conesSettings.conesSettings_MNH.conesSettings_MNH(case_path, model, nens, n_model_procs, local_commn, world_comm)[source]
Bases:
conesSettingsAbstractA class that groups settings for cones