The observation database
==========================

The high-fidelity observations :math:`\mathbf{y}_k` used by the analysis
phase (see :doc:`data_assimilation`) are not passed to CONES directly: they
are read once at start-up from a netCDF file, whose path is set by
``obsFile`` in :doc:`conesdict`. This page describes the file's expected
structure and shows two ways to build one.

Expected structure
--------------------

The file is read by
:class:`~conesToolBox.conesClasses.conesObservation.conesStaticObservation`
and the netCDF helpers in :mod:`conesToolBox.conesFunctions`
(:func:`~conesToolBox.conesFunctions.cones_count_netcdf_obs`,
:func:`~conesToolBox.conesFunctions.cones_get_netcdf_obs_coord`), which
impose the following schema:

.. list-table::
   :header-rows: 1
   :widths: 25 75

   * - Dimension
     - Meaning
   * - ``obs``
     - Number of observation points. Required, and must be named exactly
       ``obs`` — :func:`~conesToolBox.conesFunctions.cones_count_netcdf_obs`
       reads it directly as ``ds.sizes["obs"]``.
   * - ``time``
     - Number of time samples available for each observed variable.

.. list-table::
   :header-rows: 1
   :widths: 25 75

   * - Variable
     - Meaning
   * - ``x``, ``y``, ``z``
     - Coordinates of each observation point, shaped ``(obs,)``. Static in
       time: an observation is a fixed point in space.
   * - one per ``obsVar`` letter (e.g. ``u``, ``v``, ``w``)
     - The observed value of that variable at each point and time, shaped
       ``(time, obs)``. The variable names must match ``obsVar`` in
       :doc:`conesdict` exactly.
   * - the ``_std`` companion of each of the above (e.g. ``u_std``)
     - The standard deviation :math:`\sigma_{i,k}` of that observation,
       same shape. This is what fills the observation covariance
       :math:`\mathbf{R}_k = \mathrm{diag}(\sigma_{i,k}^2)` (see
       :doc:`conesdict`) and the perturbation drawn by
       :func:`~conesToolBox.conesClasses.conesEnKF.conesEnKF.setObs`.
       **Every observed variable needs one** —
       :func:`~conesToolBox.conesClasses.conesObservation.conesStaticObservation.get_vars_from_netcdf`
       looks it up unconditionally and raises a ``KeyError`` if it is
       missing.

Only the variables named in ``obsVar`` (plus their ``_std`` companion) are
actually read at run time; extra data variables in the file are ignored.

A CF-decoding gotcha
-----------------------

If you set a ``dtype`` entry directly in a variable's ``.attrs`` (rather
than its ``.encoding``) — for instance to force ``time`` to be interpreted
as ``timedelta64[ns]`` — some xarray versions refuse to CF-decode the file
on open. Both examples below set ``ds.time.attrs['dtype']`` for exactly
this reason; :func:`~conesToolBox.conesFunctions.cones_open_netcdf_dataset`
is CONES's read-side workaround, moving any such ``dtype`` attribute into
``encoding`` before decoding. Reuse it if you write your own reader.

Example 1: synthetic observations
------------------------------------

``utils/generate_observation_file.py`` builds a database of 5 point
velocity observations, constant in time — this is what the
:doc:`cavity tutorial <cones>` uses. Run it with the relative standard
deviation to apply to each value (5% here):

.. code-block:: bash

   python3 utils/generate_observation_file.py 0.05

.. literalinclude:: ../../utils/generate_observation_file.py
   :language: python
   :linenos:

Adapt the coordinates and observed values to your own case; keep every
data variable shaped ``(time, obs)`` (or ``(obs,)`` for ``x``/``y``/``z``)
and make sure every observed variable has its ``_std`` companion.

Example 2: observations from OpenFOAM probes
------------------------------------------------

If you already have a reference/high-fidelity run, ``utils/probes_to_observation_netcdf.py``
converts the output of OpenFOAM's ``probes`` function object (as configured
in ``system/probes``) into an observation database, reading the probe
coordinates from the file's own header comments:

.. code-block:: bash

   python3 utils/probes_to_observation_netcdf.py postProcessing/probes/0/U

It currently hardcodes ``num_obs = 5`` and a flat 5% relative standard
deviation — open the script and adjust both to match your ``system/probes``
setup before relying on it.