Databases (db)

class rubin_sim.maf.db.DisplayRow(**kwargs)

Bases: Base

Define contents and format of the displays table.

(Table to list the display properties for each metric.)

class rubin_sim.maf.db.MetricRow(**kwargs)

Bases: Base

Define contents and format of metric list table.

(Table to list all metrics, their info_label, and their output data files).

class rubin_sim.maf.db.PlotRow(**kwargs)

Bases: Base

Define contents and format of plot list table.

(Table to list all plots, link them to relevant metrics in MetricList, and provide info on filename).

class rubin_sim.maf.db.ResultsDb(out_dir=None, database=None, verbose=False, timeout=180)

Bases: object

ResultsDb is a sqlite database containing information on the metrics run via MAF, the plots created, the display information (such as captions), and any summary statistics output.

static build_summary_name(metric_name, metric_info_label, slicer_name, summary_stat_name=None)

Standardize a complete summary metric name, combining the metric + slicer + summary + info_label.

close()

Close connection to database.

downgrade_database()

Downgrade resultsDb to work with v0.10<MAF< v1.0 There is also a ‘upgradeDatabase’ to update to the newer style with ‘metric_info_label.

get_all_metric_ids()

Return a list of all metric_ids.

get_metric_data_files(metric_id=None)

Get the metric data filenames for all or a single metric. Returns a list.

get_metric_display_info(metric_id=None)

Get the contents of the metrics and displays table, together with the ‘basemetric_name’ (optionally, for metric_id list). Returns a numpy array of the metric information + display information.

An underlying assumption here is that all metrics have some display info. This may not always be the case.

get_metric_id(metric_name, slicer_name=None, metric_info_label=None, run_name=None)

Find metric bundle Ids from the metric table.

Parameters:

• metric_name (str) – Name of the Metric

• slicer_name (str, opt) – Name of the Slicer to match

• metric_info_label (str, opt) – Metadata value to match

• run_name (str, opt) – Name of the simulation (run_name) to match

Returns:

metric_id – List of matching metric_ids

Return type:

list of int

get_metric_id_like(metric_name_like=None, slicer_name_like=None, metric_info_label_like=None, run_name=None)

Find metric bundle Ids from the metric table, but search for names ‘like’ the values. (instead of a strict match from get_metric_id).

Parameters:

• metric_name (str) – Partial name of the Metric

• slicer_name (str, opt) – Partial name of the Slicer to match

• metric_info_label (str, opt) – Partial info_label value to match

• run_name (str, opt) – Name of the simulation (run_name) to match (exact)

Returns:

metric_id – List of matching metric_ids

Return type:

list of int

get_metric_info(metric_id=None, with_sim_name=False)

Get the simple metric info, without display information.

Parameters:

• metric_id (int list, or None) – If None, data for all metrics are returned. Otherwise, only data corresponding to the supplied metric ID or IDs are returned

• with_sim_name (bool) – If True, include the run name in the fields returned

Returns:

plotFiles –

metric_id

The metric ID

metric_name

The metric name

basemetric_names

The base metric names

slicer_name

The name of the slicer used in the bundleGroup

sql_constraint

The full sql constraint used in the bundleGroup

metric_info_label

Metadata extracted from the sql_constraint (usually the filter)

metric_datafile

The file name of the file with the metric data itself.

run_name

The name of the run plotted (if with_sim_name was True)

Return type:

numpy.recarray

get_plot_files(metric_id=None, with_sim_name=False)

Find the metric_id, name, info_label, and all plot info (optionally for metric_id list).

Parameters:

• metric_id (int list, or None) – If None, plots for all metrics are returned. Otherwise, only plots corresponding to the supplied metric ID or IDs are returned

• with_sim_name (bool) – If True, include the run name in the fields returned

Returns:

plotFiles –

metric_id

The metric ID

metric_name

The metric name

metric_info_label

info_label extracted from the sql constraint (usually the filter)

plot_type

The plot type

plot_file

The full plot file (pdf by default)

thumb_file

A plot thumbnail file name (png)

run_name

The name of the run plotted (if with_sim_name was True)

Return type:

numpy.recarray

get_run_name()

Return a list of the run_names for the metric bundles in the database.

get_summary_stats(metric_id=None, summary_name=None, summary_name_like=None, summary_name_notlike=None, with_sim_name=False)

Get the summary stats (optionally for metric_id list). Optionally, also specify the summary metric name. Returns a numpy array of the metric information + summary statistic information.

Parameters:

• metric_id (int or list of int) – Metric bundle Ids to match from the metric table

• summary_name (str, opt) – Match this summary statistic name exactly.

• summary_name_like (str, opt) – Partial match to this summary statistic name.

• summary_name_notlike (str, opt) – Exclude summary statistics with summary names like this.

• with_sim_name (bool, opt) – If True, add the run_name to the returned numpy recarray.

Returns:

summarystats – Numpy recarray containing the selected summary stat information.

Return type:

np.recarray

open()

Open connection to database

update_database()

Update the results_db from ‘metricMetaData’ to ‘metric_info_label’ and now also changing the camel case to snake case (metricId to metric_id, etc.).

This updates results_db to work with the current version of MAF, including RunComparison and showMaf. There is also a ‘downgrade_database’ to revert to the older style with ‘metricMetadata.

update_display(metric_id, display_dict, overwrite=True)

Add a row to or update a row in the displays table.

Parameters:

• metric_id (int) – The metricID for this metric bundle in the metrics table

• display_dict (dict) – Dictionary containing the display info (group/subgroup/order/caption)

• overwrite (bool, opt) – Replaces existing row with same metric_id if overwrite is True.

update_metric(metric_name, slicer_name, run_name, sql_constraint, metric_info_label, metric_datafile)

Add a row to or update a row in the metrics table.

Parameters:

• metric_name (str) – Name of the Metric

• slicer_name (str) – Name of the Slicer

• run_name (str) – Name of the simulation (run_name, simName, run_name..)

• sql_constraint (str) – sql_constraint relevant for the metric bundle

• metric_info_label (str) – Information associated with the metric. Could be derived from the sql_constraint or could be a more descriptive version, specified by the user.

• metric_datafile (str) – The data file the metric bundle output is stored in.

Returns:

• metric_id (int) – The Id number of this metric in the metrics table.

• If same metric (same metric_name, slicer_name, run_name,

• sql_constraint, infoLabel) already exists, it does nothing.

update_plot(metric_id, plot_type, plot_file, overwrite=False)

Add a row to or update a row in the plot table.

Parameters:

• metric_id (int) – The metric Id of this metric bundle in the metrics table

• plot_type (str) – The type of this plot (oneDbinned data, healpix map, etc.)

• plot_file (str) – The filename for this plot

• overwrite (bool) – If True, replaces existing row. If False, an additional plot is added to the output (e.g. with a different range of color values, etc).

update_summary_stat(metric_id, summary_name, summary_value, ntry=3, pause_time=100)

Add a row to or update a row in the summary statistic table.

Most summary statistics will be a simple name (string) + value (float) pair. For special summary statistics which must return multiple values, the base name can be provided as ‘name’, together with a np.ndarray as ‘value’, where the array also has ‘name’ and ‘value’ columns (and each name/value pair is then saved as a summary statistic associated with this same metric_id).

Parameters:

• metric_id (int) – The metric Id of this metric bundle

• summary_name (str) – The name of this summary statistic

• summary_value (: float or np.ndarray) – The value for this summary statistic. If this is a np.ndarray, then it should also have ‘name’ and ‘value’ columns to save each value to rows in the summary stats.

• ntry (int, opt) – The number of times to retry if database is locked.

• pause_time (int, opt) – Time to wait until trying again.

class rubin_sim.maf.db.SummaryStatRow(**kwargs)

Bases: Base

Define contents and format of the summary statistics table.

(Table to list and link summary stats to relevant metrics in MetricList, and provide summary stat name, value and potentially a comment).

class rubin_sim.maf.db.TrackingDb(database=None, trackingDbverbose=False)

Bases: object

Sqlite database to track MAF output runs and their locations, for show_maf

add_run(run_group=None, run_name=None, run_comment=None, run_version=None, run_date=None, maf_comment=None, maf_version=None, maf_date=None, maf_dir=None, db_file=None, maf_run_id=None)

Add a run to the tracking database.

Parameters:

• run_group (str, optional) – Set a name to group this run with (eg. “Tier 1, 2016”).

• run_name (str, optional) – Set a name for the opsim run.

• run_comment (str, optional) – Set a comment describing the opsim run.

• run_version (str, optional) – Set the version of opsim.

• run_date (str, optional) – Set the date the opsim run was created.

• maf_comment (str, optional) – Set a comment to describe the MAF analysis.

• maf_version (str, optional) – Set the version of MAF used for analysis.

• maf_date (str, optional) – Set the date the MAF analysis was run.

• maf_dir (str, optional) – The relative path to the MAF directory.

• db_file (str, optional) – The relative path to the Opsim SQLite database file.

• maf_run_id (int, optional) – The maf_run_id to assign to this record in the database (note this is a primary key!). If this run (ie the maf_dir) exists in the database already, this will be ignored.

Returns:

maf_run_id – The maf_run_id stored in the database.

Return type:

int

delRun(runId)

Remove a run from the tracking database.

rubin_sim.maf.db.add_run_to_database(maf_dir, tracking_db_file, run_group=None, run_name=None, run_comment=None, maf_comment=None, db_file=None, maf_version=None, maf_date=None, sched_version=None, sched_date=None, skip_extras=False)

Adds information about a MAF analysis run to a MAF tracking database.

Parameters:

• maf_dir (str) – Path to the directory where the MAF results are located.

• tracking_db_file (str) – Full filename (+path) to the tracking database to use.

• run_group (str, optional) – Name to use to group this run with other opsim runs.

• run_name (str, optional) – Name of the opsim run.

• run_comment (str, optional) – Comment about the opsim run.

• run_version (str, optional) – Value to use for the opsim version information.

• maf_comment (str, optional) – Comment about the MAF analysis.

• db_file (str, optional) – Relative path + name of the opsim database file.