HiPERCAM commands

This page documents the pipeline commands which are all part of hipercam.scripts. Help on any given command can be obtained at the terminal using e.g. pydoc hipercam.scripts or pydoc hipercam.scripts.rtplot. This is often the most useful way to get information quickly. If you are new to the pipeline, it would be worth reading the Parameter specification section on how to call the pipeline commands and specify parameters.

Main commands

This section lists the main HiPERCAM commands 1 ; see also the bottom of the page for all the documentation on the commands in one long list. Clicking on a command name in the table below will take you to the relevant section of this long list. The table also indicates contexts in which each command is particularly useful.

Command

Purpose

Observing

Reduction

Plots & analysis

Arithematic

Information

add

add two frames

Yes

averun

average a series of frames in a run

Yes

Yes

cadd

add a constant to a frame

Yes

cdiv

divide a frame by a constant

Yes

cmul

multiply a frame by a constant

Yes

combine

combine a list of frames

Yes

Yes

Yes

csub

subtract a constant from a frame

Yes

div

divide one frame by another

Yes

fits2hcm

convert foreign FITs files to hcm format

Yes

flagcloud

flag cloudy and junk data in a reduce log

Yes

Yes

ftargets

automatically find targets using “sep”

Yes

Yes

genred

create a reduce file

Yes

Yes

grab

split frames out of a run

Yes

Yes

hfilter

filter a HiPERCAM image

Yes

hinfo

lists information on a HiPERCAM image

Yes

hist

plot a histogram of a frame

Yes

Yes

hlog2col

produce ASCII column data from a reduce log

Yes

Yes

Yes

hlog2fits

convert a reduction log file to FITS

Yes

Yes

hls

list the runs on the HiPERCAM server

Yes

Yes

hplot

plot a frame

Yes

Yes

Yes

ltimes

list times of a run

Yes

makebias

combine a run to make a bias frame

Yes

Yes

makedark

combine a run to make a dark frame

Yes

makeflat

combine a list of frames into a flat

Yes

mstats

list stats of multiple frames from a run

Yes

mul

multiply two frames

Yes

rtplot

plot frames as they come in [matplotlib]

Yes

Yes

Yes

plog

plot output log from reduce

Yes

Yes

redanal

analyse a reduction log file

Yes

Yes

reduce

carry out photometric reduction

Yes

Yes

rtplot

plot frames as they come in [pgplot]

Yes

Yes

Yes

rupdate

updates old reduce files

Yes

Yes

setaper

define the photometric apertures

Yes

Yes

setdefect

define a file of CCD defects

Yes

splice

splice two frames together

Yes

stats

report statistics of a frame

Yes

sub

subtract two frames

Yes

uls

list the runs on the ULTRACAM server

Yes

Parameter specification

The pipeline commands are distinct from standard unix commands in having a ‘memory’, which is implemented through storage of inputs in disk files for each command, and also in prompting you if you don’t specify a parameter on the command line. Parameters can be ‘global’ or ‘local’ depending upon whether they are reset across multiple commands or just the command of interest. The parameter memory, along with the use of backslashes ‘\’ to accept default values can save a huge amount of typing making for efficient operation once you get up to speed.

Basic parameter input

The command rtplot has more parameters than most others and is a good one to start with. Suppose then that we have a raw HiPERCAM file, run0076.fits, that we want to plot. If we type ‘rtplot’ and follow the prompts, the first few lines might be:

rtplot
run - run name [run0064]: run0076
first - first frame to plot [10]: 1

This shows that the last time rtplot was invoked, it was used to look as run0064.fits starting with frame 10. Note that the extension ‘.fits’ is not needed: HiPERCAM makes significant use of extensions to differentiate between different forms of file, all of which could be associated with the same run and therefore start with the ‘run0064’ or whatever. Alternatively we could have typed:

rtplot run0076
first - first frame to plot [10]: 1

which would accomplish the same, setting the run name according to its position in the command-line arguments. For this to work, you need to be certain of the ordering. If you are not sure of the order, but you are sure that the parameter is called run, then:

rtplot run=run0076
first - first frame to plot [10]: 1

will do, and the first two parameters could be similarly specified:

rtplot first=1 run=run0076

Note that by naming the parameters, the order becomes immaterial. Now, assuming that the command was completed, if you run rtplot again you might get:

rtplot
run - run name [run0076]:
first - first frame to plot [1]:
ccd - CCD(s) to plot [0 for all] [3]: 3 4
nx - number of panels in X [2]:
bias - bias frame ['none' to ignore] [none]:
msub - subtract median from each window? [False]: True

You will see that the defaults for run and first have been updated by the previous invocations of the command. This the parameter “memory” referred to earlier. Now suppose that we wish to repeat the command without changing any parameter. Then a simple:

rtplot \\

will do. The double backslash ‘\\’ indicates “take the default value for all remaining parameters”. Two are needed because the shell will strip one of them. If you let at least one prompting line be output then a single backslash will accomplish the same thing:

rtplot
run - run name [run0076]: \

If you wish to change just one parameter, say msub, then much typing can be saved with:

rtplot msub=no \\

(NB: Boolean True/False parameters like msub can be reset False with either ‘no’, ‘false’ or even just ‘n’.) Combined with up and down arrow keys, and commands can be quickly repeated or modified.

Hidden parameters and special keywords

A look through the Command definitions will reveal that many commands have “hidden” parameters that are not usually prompted for. The idea behind this is to reduce the level of prompting, particularly for those parameters that hardly ever change. The values of all parameters can be revealed through the use of a special keyword list`:

rtplot list \\
rtplot
source = hl
device = 1/xs
width = 0.0
height = 0.0
run = run0076
first = 1
twait = 10.0
tmax = 20.0
ccd = 3 4
nx = 2
pause = 0.0
plotall = False
bias = none
msub = True
iset = p
plo = 10.0
phi = 99.7
xlo = 600.0
xhi = 950.0
ylo = 100.0
yhi = 400.0

This reveals parameters source, device, width and height that were never prompted for in the commands of the previous section. Their values can be changed by giving another special keyword prompt:

rtplot prompt source - data source [hs, hl, us, ul, hf] [hl]: device - plot device [1/xs]: width - plot width (inches) [0.0]: 10 height - plot height (inches) [0.0]: 8 run - run name [run0076]:

etc. Usually once changed in this way, the hidden parameters will keep the new values, although there are some which will return to a standard default on the basis that it is almost always what is wanted and it would be dangerous to default to a different value.

In addition to list and prompt, there is a third special keyword nodefs which means do not read or write any default parameter values from files. If nodefs is specified, then all parameters values need to be spelt out; it’s use is inside scripts in circumstances where multiple instances are being used, or when there is interactive work going on too, to avoid causing problems by over-writing the default files. Avoid using these keywords in any other context. e.g. you would be asking for trouble if you named your files list.hcm or prompt.hcm.

Strings with spaces

If you need to specify a string with a space in it, either let a command prompt you and type as normal, or quote it on the command line, e.g.:

rtplot run="silly file name" \\

Setting a string to be blank

If you want to set a string to a blank, then a carriage return won’t work when prompted for it since this will just keep whatever default is set. Instead type ‘’ (two single quotes) which will be interpreted as implying an empty string.

Parameter ranges

Many numerical parameters have mimimum and/or maximum values set. Typing ‘?’ at the prompt should tell you what the values are. Typing ‘min’ or ‘max’ will set the parameter to have the appropriate value.

Tab completion

File name input is helped by tab completion: start typing, hit <tab> and if the file exists, the name might be completed.

Global vs local parameters

All HiPERCAM pipeline parameters fall into one of two classes, either being ‘local’ to a command or ‘global’ to multiple commands. The run parameter of rtplot for instance also appears in grab and if you change it in rtplot, it will be changed in grab. This is very useful when running a series of commands on the same file as the commands almost ‘know’ what you want, saving much typing.

Trouble-shooting parameter input

It very rarely happens that the file I/O needed to read the default values can get confused. This can happen e.g. if the data type of a parameter has changed. If very odd things seems to happen when you try to start a command, then you might want to track down where the default files are located. Usually this will be in .hipercam in your home directory, unless the environment variable HIPERCAM_ENV has been defined to re-direct where the default files are stored. Once you have located them it is always safe to delete one or more or all of the default files (end with .def). The worst that happens is that the commands have lost the default values.

Errors reported by scripts

If you use the pipeline for any significant time, you will at some point get a rather forbidding-looking error traceback such as:

reduce run=run013 \\
Traceback (most recent call last):
  File "/storage/astro1/phsaap/software/python/bin/reduce", line 11, in <module>
    load_entry_point('hipercam==0.19.9.dev22+g758df9d.d20200303', 'console_scripts', 'reduce')()
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 405, in reduce
    with spooler.data_source(source, resource, first, full=False) as spool:
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/spooler.py", line 323, in data_source
    return UcamDiskSpool(resource, first)
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/spooler.py", line 108, in __init__
    self._iter = ucam.Rdata(run, first, False)
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/ucam.py", line 637, in __init__
    Rhead.__init__(self, run, server)
  File "/storage/astro1/phsaap/software/python/lib/python3.6/site-packages/hipercam/ucam.py", line 152, in __init__
    udom = xml.dom.minidom.parse(run + '.xml')
  File "/warwick/desktop/2018/software/MPI/GCC/7.3.0-2.30/OpenMPI/3.1.1/Python/3.6.6/lib/python3.6/xml/dom/minidom.py", line 1958, in parse
    return expatbuilder.parse(file)
  File "/warwick/desktop/2018/software/MPI/GCC/7.3.0-2.30/OpenMPI/3.1.1/Python/3.6.6/lib/python3.6/xml/dom/expatbuilder.py", line 910, in parse
    with open(file, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'run013.xml'

Yikes! Although it looks awful, it simply reflects the chain of function calls that led to the problem, an extremely useful diagnostic of problems in Python code. Such tracebacks look a bit ugly, but almost certainly, in most cases, including this one, they are caused by incorrect parameter inputs. The one to look at is probably the last line or two, which reveals in this case that an expected file ‘run013.xml’ was not found; a directory listing would confirm this.

Such errors follow from a standard Python approach of not trying to add endless checking code, but to let the code tell you what happened when errors are encountered. This has the merit of being very informative (and de-clutters code), but it can make it hard to distinguish between an essentially trivial issue, such as a missing file, and a genuine problem with the code. If however you find you can’t get round a problem and the error reported does not look innocent, then it might be a time to report the problem. We will do our best to provide fast solutions to critical issues.

Command definitions

This section contains documentation auto-generated from the code. This is the same as is returned from clicking command names in the lists at the top of this page or from using pydoc in a terminal (e.g. try pydoc hipercam.scripts.reduce). Each command appears as a function (an implementation detail), followed by a highlighted line showing the parameters one can use on the command-line. Inputs in square brackets such as [source] are hidden by default; those in round brackets e.g. (plot) may or may not be prompted depending upon earlier inputs. It is always safest when first running a command simply to type its name and hit enter and let the command itself prompt you for input. Many commands have hidden parameters that can only be revealed by typing e.g. rtplot prompt. These are usually parameters that rarely need changing, but you are sure sometimes to need to alter them. See the Parameter specification section for details on how to specify command parameters.

In the one-line descriptions below, run refers to a complete run, containing multiple images, stored in a .fits file. frame refers to a single image from a run as might be extracted using grab. These have file extension ‘.hcm’ to distinguish them, although they are also FITS-format files.

hipercam.scripts.add(args=None)

add input1 input2 [ccd win] output

Adds two hcm frames and outputs the result. Can be applied only to particular CCDs and windows if wanted.

Parameters:

input1string

first input hcm file

input2string

second input hcm file to add to the first.

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as either input1 or input2 in which case the input file will be over-written.

hipercam.scripts.averun(args=None)

averun [source] (run first last twait tmax | flist) trim ([ncol nrow]) bias dark flat [method sigma adjust clobber] output

Averages images from a run using median combination, skipping the junk frames that result from NSKIP / NBLUE options in HiPERCAM and ULTRACAM data. averun is meant to be a simple tool to create median frames suitable prior to aperture selection with setaper. See combine if you want more fine-grained control over frame averaging. (averun uses a combination of grab [if needed] and combine).

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : ULTRACAM server
‘ul’ : local ULTRACAM .xml/.dat files
‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for ``source’’ can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststr [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame (‘0’ is not supported).

lastint [if source ends ‘s’ or ‘l’]

last exposure number must be >= first.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

trimbool

True to trim columns and/or rows off the edges of windows nearest the readout. Useful for ULTRACAM particularly.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

darkstr

Name of dark frame to subtract, ‘none’ to ignore.

flatstr

Name of flat field to divide by, ‘none’ to ignore.

methodstr [hidden, defaults to ‘m’]

‘m’ for median, ‘c’ for clipped mean. See below for pros and cons.

sigmafloat [hidden; if method == ‘c’]

With clipped mean combination, pixels that deviate by more than sigma RMS from the mean are kicked out. This is carried out in an iterative manner. sigma <= 0 implies no rejection, just a straight average. sigma=3 is typical.

adjuststr [hidden; defaults to ‘i’]

adjustments to make: ‘i’ = ignore; ‘n’ = normalise the mean of all frames to match the first; ‘b’ = add offsets so that the mean of all frames is the same as the first. Option ‘n’ is useful for twilight flats; ‘b’ for combining biases.

clobberbool [hidden]

clobber any pre-existing output files

outputstr

output file. Set by default to match the last part of “run” (but it will have a different extension so they won’t clash)

hipercam.scripts.cadd(args=None)

cadd input const [ccd win] output

Adds a constant to a HiPERCAM hcm frame. Can be applied only to particular CCDs and windows if wanted.

Parameters:

inputstring

input hcm file name

constfloat

constant to add

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as input in which case the file will be over-written.

hipercam.scripts.cdiv(args=None)

cdiv input const [ccd win] output

Divides a HiPERCAM hcm frame by a constant. Can be applied only to particular CCDs and windows if wanted.

Parameters:

inputstring

input hcm file name

constfloat

constant to divide by

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as input in which case the file will be over-written.

hipercam.scripts.cmul(args=None)

cmul input const [ccd win] output

Multiplies a HiPERCAM hcm frame by a constant. Can be applied only to particular CCDs and windows if wanted.

Parameters:

inputstring

input hcm file name

constfloat

constant to multiply by

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as input in which case the file will be over-written.

hipercam.scripts.combine(args=None)

combine list bias dark flat method (sigma) adjust (usemean) [plot clobber] output

Combines a series of images defined by a list using median or clipped mean combination. Only combines those CCDs for which is_data() is true (i.e. it skips blank frames caused by NSKIP / NBLUE options)

Parameters:

liststring

list of hcm files with images to combine. The formats of the images should all match

biasstring

Name of bias frame to subtract, ‘none’ to ignore.

darkstring

Name of dark frame to subtract, ‘none’ to ignore.

flatstring

Name of flat field frame to subtract, ‘none’ to ignore.

methodstring

‘m’ for median, ‘c’ for clipped mean. See below for pros and cons.

sigmafloat [if method == ‘c’]

With clipped mean combination, pixels that deviate by more than sigma RMS from the mean are kicked out. This is carried out in an iterative manner. sigma <= 0 implies no rejection, just a straight average. sigma=3 is typical.

adjuststring

adjustments to make: ‘i’ = ignore; ‘n’ = normalise the mean or median of all frames to match the first; ‘b’ = add offsets so that the mean or median of all frames is the same as the first. Option ‘n’ is useful for twilight flats and fringe frames; ‘b’ is good for combining biases.

usemeanbool [if adjust == ‘n’ or ‘b’]

True to use the mean rather than the median for normalisation or biass offsetting.

plotbool [hidden, if adjust == ‘n’ or ‘b’; defaults to False]

make a plot of the mean versus frame number. This can provide a quick check that the frames are not too different.

clobberbool [hidden]

clobber any pre-existing output files

outputstring

output file

Clipped mean can work well for large numbers of frames but gets worse for small numbers as the RMS can be heavily influenced by a single bad value. The median can be better in such cases, but has the downside of digitisation noise. For instance, the average of 100 bias frames could have a noise level significantly below 1 count, depending upon the readout noise, and the +/- 0.5 count uncertainty of median combination may be worse than this.

Note

This routine reads all inputs into memory, so can be a bit of a hog. However, it does so one CCD at a time to alleviate this. It will fail if it cannot find a valid frame for any CCD.

hipercam.scripts.csub(args=None)

csub input const [ccd win] output

Subtracts a constant from a HiPERCAM hcm frame. Can be applied only to particular CCDs and windows if wanted.

Parameters:

inputstring

input hcm file name

constfloat

constant to subtract

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as input in which case the file will be over-written.

hipercam.scripts.div(args=None)

div input1 input2 [ccd win] output

Divides two hcm frames and outputs the result. Can be applied only to particular CCDs and windows if wanted.

Parameters:

input1string

first input hcm file

input2string

second input hcm file to divide into the first.

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as either input1 or input2 in which case the input file will be over-written.

hipercam.scripts.fits2hcm(args=None)

fits2hcm flist origin

This routine converts “foreign” data into a format suitable for the pipeline. HiPERCAM’s hcm files are in fact FITS-format so this is mostly a case of re-organising the files.

Parameters:

fliststr

name of list of input FITS-format files. ‘flist’ should end ‘.lis’. The output file names will have the same rootname as the input files (but without any leading directories) but end in ‘.hcm’

originstr

origin of the data. Currently recognised:

HICKS :

University of Sheffield 16” Hicks Telescope SBIG ST10-XME CCD

INTWFC :

Wide field Camera on the INT. Just operates on a single CCD’s-worth of data.

LCOGT :

Las Cumbres Observatory, based on some FTN data taken in 2019.

LTIO :

Liverpool Telescope IO camera.

LTRISE :

Liverpool Telescope RISE camera. I don’t know how general my code is in this case. I assume XBIN=YBIN=2.

PT5M :

University of Sheffield/Durham pt5m telescope on La Palma QSI 532 CCD

ROSA :

University of Sheffield 10” Hicks Telescope ATIK ONE 6.0 CCD

WHTCAM :

Camera used by Richard Ashley in Feb 2021.

overwritebool

overwrite files on output

hipercam.scripts.flagcloud(args=None)

flagcloud hlog aper1 aper2 ccd delta output

Interactive flagging of cloud-affected or otherwise bad points in a HiPERCAM log file. You either mark a range of times as cloudy, or individual points as junk. If you mark a range, then all apertures of all CCDs will be flagged with the bitmask value CLOUDS. Individual points will be flagged as JUNK. Note that nothing is done to the data apart from changing the bitmask flags, so it is then up to you to test for these later on. This also means the flagging operation is easily reversible by clearing the CLOUD and JUNK flags, e.g. by AND-ing the flags with ~JUNK and ~CLOUDS. It is also possible to flag individual points as CLOUDS but these will not propagate across CCDs so it is probably not advisable to do this. Similarly range can be flagged as JUNK although it is unusual for all apertures of all CCDs to be junk, although bad seeing / awful cloud can cause this.

Junk points are marked red, cloudy points orange. OK aperture 1 points are plotted green, aperture 2 blue, and their ratio black. What is meant by ‘junk’ as opposed to ‘cloud’ is really down to the user. I tend to reserve junk for one-off points affected by bad cosmic rays and satellites, but in general it is probably sensible to think of “junk” as points you never want to see again versus “clouds” meaning data that you might want either to mask or use down the line, but would would like some way to know that it was affected by clouds. Some genuine “cloudy” data will be so bad that it will be better flagged as junk however. You can flag the same point as both “cloud” and “junk”, but “junk” is the stronger condition.

Bitmasks propagate when data are combined so a point flagged junk in aperture 2 but not aperture 1 will be flagged junk in the ratio of 1 divided by 2. You can also recover points in this routine; doing so will clear both their junk and/or cloud status. At the moment this is only possible on a point-by-point basis.

Interaction is via the cursor and hitting specific keys. Common options are lower case; less common ones upper case. Hitting the X on the plot will abort without saving the results. ‘q’ to quit saves the results. ‘h’ gives further help on the options.

Parameters:

hlogstr

ASCII log file, as produced by reduce.

aper1str

the name of first aperture to look at

aper2str

the name of second aperture to look at. The ratio aper1 / aper2 will be plotted along with the two separately, scaled by their maximum, all in the same panel.

ccdstr

CCD(s) to plot, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’, or even ‘1 2 3’ are possible inputs (without the quotes). Note the space separation when multiple CCDs are specified. If you want to plot more than one CCD, then you will get multiple panels in the Y direction, but their X-axes are kept in lock step when panning or zooming.

deltafloat

separation to use to space the plots in a given panel, each of which is normalised to 1. A value of 1 is recommended because then the second aperture should end with a typical level of 0, and any dips below this show the extent of the cloud. e.g. -0.9 would suggest an approximate 90% loss of flux due to cloud, ignoring extinction.

outputstr

name of modified version of the Hlog for output. Can overwrite the original if you dare.

hipercam.scripts.ftargets(args=None)

ftargets [source device width height] (run first [twait tmax] | flist) trim ([ncol nrow]) (ccd (nx)) [pause] thresh fwhm minpix output bias flat msub iset (ilo ihi | plo phi) xlo xhi ylo yhi

This script carries out the following steps for each of a series of images:

  1. detects the sources,

  2. identifies isolated targets suited to profile fits,

  3. fits 2D Moffat profiles to these,

  4. Saves results to disk.

The profile fits are carried out because sep does not return anything that can be used reliably for a FWHM.

Several parameters depends on the object detection threshold retuned by the source detection. This is referred to as threshold. The source detection is carried out using sep which runs according to the usual source extractor algorithm of Bertin.

The script plots the frames, with ellipses at 3*a,3*b indicated in red, green boxes indicating the range of pixels identified by sep, and blue boxes marking the targets selected for FWHM fitting (the boxes indicate the fit region).

Parameters:

sourcestring [hidden]

Data source, five options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : ULTRACAM server
‘ul’ : local ULTRACAM .xml/.dat files
‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for ``source’’ can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

devicestring [hidden]

Plot device. PGPLOT is used so this should be a PGPLOT-style name, e.g. ‘/xs’, ‘1/xs’ etc. At the moment only ones ending /xs are supported.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

runstring [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststring [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame; set = 0 to always try to get the most recent frame (if it has changed). For data from the HiPERCAM server, a negative number tries to get a frame not quite at the end. i.e. -10 will try to get 10 from the last frame. This is mainly to sidestep a difficult bug with the acquisition system.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

trimbool [if source starts with ‘u’]

True to trim columns and/or rows off the edges of windows nearest the readout which can sometimes contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

ccdstring

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc.

nxint [if more than 1 CCD]

number of panels across to display.

pausefloat [hidden]

seconds to pause between frames (defaults to 0)

threshfloat

threshold (mutiple of RMS) to use for object detection. Typical values 2.5 to 4. The higher it is, the fewer objects will be located, but the fewer false detections will be made.

fwhmfloat

FWHM to use for smoothing during object detection. Should be comparable to the seeing.

minpixint

Minimum number of pixels above threshold before convolution to count as a detection. Useful in getting rid of cosmics and high dark count pixels.

rminfloat

Closest distance of any other detected object for an attempt to be made to fit the FWHM of an object [unbinned pixels].

pminfloat

Minimum peak height for an attempt to be made to fit the FWHM of an object. This should be a multiple of the object detection threshold (returned by sep for each object).

pmaxfloat

Maximum peak height for an attempt to be made to fit the FWHM of an object. Use to exclude saturated targets [counts]

emaxfloat

Maximum elongation (major/minor axis ratio = a/b), > 1. Use to reduce very non-stellar profiles.

nmaxint

Maximum number of FWHMs to measure. Will take the brightest first, judging by the flux.

biasstring

Name of bias frame to subtract, ‘none’ to ignore.

flatstring

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

output: string

Name of file for storage of results. Will be a fits file, with results saved to the HDU 1 as a table.

isetstring [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD basis.

ilofloat [if iset=’d’]

lower intensity level

ihifloat [if iset=’d’]

upper intensity level

plofloat [if iset=’p’]

lower percentile level

phifloat [if iset=’p’]

upper percentile level

xlofloat

left-hand X-limit for plot

xhifloat

right-hand X-limit for plot (can actually be < xlo)

ylofloat

lower Y-limit for plot

yhifloat

upper Y-limit for plot (can be < ylo)

hipercam.scripts.genred(args=None)

genred apfile rfile comment bias flat dark linear [inst skipbadt ncpu extendx ccd location smoothfwhm method beta betamax fwhm fwhmmin fwhmmax searchwidth thresh hminref hminnrf rfac rmin rmax sinner souter scale psfgfac psfwidth psfpostweak]

Generates a reduce file as needed by reduce or psf_reduce. You give it the name of an aperture file and a few other parameters and it will write out a reduce file which you can then refine by hand. A few simplifying assumptions are made, e.g. that the target is called ‘1’; see below for more. This script effectively defines the format of reduce files. The script attempts above all to generate a self-consistent reduce file. e.g. if there are no apertures in CCD 5, it does not attempt to plot any corresponding light curves.

To avoid excessive prompting, genred has many hidden parameters. The very first time you use it on a run, specify prompt on the command line to see all of these. They are chosen to be the parameters most likely to vary with telescope or conditions; many others are left at default values and require editing to change. If you find yourself repeatedly editing a parameter, let me know and I will add it to this routine.

Parameters:

apfilestring

the input aperture file created using setaper (default extension .ape). This will be read for the targets. The main target will be assumed to have been called ‘1’, the main comparison ‘2’. If there is a ‘3’ it will be plotted relative to ‘2’; all others will be ignored for plotting purposes. Target ‘2’ will be used to define the position and transmission plots for one CCD only [user definable]. Target ‘1’ will be used for the seeing plot unless it is linked when target ‘2’ will be used instead.

rfilestring

the output reduce file created by this script. The main target will be assumed to have been called ‘1’, the main comparison ‘2’. If there is a ‘3’ it will be plotted relative to ‘2’; all others will be ignored for plotting purposes.

commentstring

comment to add near the top of the reduce file. Obvious things to say are the target name and the name of the observer for instance.

biasstring

Name of bias frame; ‘none’ to ignore.

flatstring

Name of flat field frame; ‘none’ to ignore.

darkstring

Name of dark frame; ‘none’ to ignore.

linearstring

light curve plot linear (else magnitudes)

inststring [hidden]

the instrument (needed to set nonlinearity and saturation levels for warning purposes. Possible options listed.

skipbadtbool [hidden]

whether to skip data with bad times or not. Should be True for ULTRACAM and ULTRASPEC; False for HiPERCAM.

ncpuint [hidden]

some increase in speed can be obtained by running the reduction in parallel. This parameter is the number of CPUs to use. The parellisation is over the CCDs, and there is no point having ncpu greater than the number of CCDs, but it should ideally equal the number of CCDs. There is no point using this for single CCD reduction.

ngroupint [hidden, if ncpu > 1]

to reduce parallelisation overheads, this parameter means that ngroup frames are read before being split up for the parallisation step is applied.

extendxfloat [hidden]

how many minutes to extend light curve plot by at a time

ccdstring [hidden]

label of the (single) CCD used for the position plot

locationstring [hidden]

whether to reposition apertures or leave them fixed.

toffsetint [hidden]

integer offset to subtract from the MJD times in order to reduce round-off. i.e. rather 55678.123456789, if you specified toffset=55600, you would reduce the round-off error by a factor ~1000. With typical MJDs, round off is around the 0.5 microsecond level. If you want to improve on that, set an appropriate offset value. It is set by default to 0, i.e. it will be 0 unless you explicitly set it otherwise. The value used is recorded in the log file. toffset must be >= 0.

smoothfwhmfloat [hidden]

FWHM to use for smoothing during initial search [binned pixels]

fftbool [hidden]

whether or not to use FFTs when carrying out the convolution operation used in the initial search. No effect on results, but could be faster for large values of smoothfwhm.

methodstring

profile fitting method. ‘g’ for gaussian, ‘m’ for moffat

betafloat [hidden]

default Moffat exponent to use to start fitting

betamaxfloat [hidden]

maximum Moffat exponent to pass on to subsequent fits. Prevents it wandering to silly values which can happen.

fwhmfloat [hidden]

the default FWHM to use when fitting [unbinned pixels].

fwhmminfloat [hidden]

the minimum FWHM to allow when fitting [unbinned pixels].

fwhmmaxfloat [hidden]

the maximum FWHM to allow when fitting [unbinned pixels].

searchwidthint [hidden]

half width in (binned) pixels for the target searches

fitwidthint [hidden]

half width in (binned) pixels for the profile fits

maxshiftfloat [hidden]

maximum shift of non-reference targets relative to the initial positions derived from the reference targets. The reference targets should give good initial positions, thus this can be set to quite a small value to improve robustness against duff positions, caused e.g. by cosmic rays or field crowding in periods of bad seeing. Use linked apertures for more difficult cases still.

threshfloat [hidden]

RMS rejection threshold for profile fits.

hminreffloat [hidden]

minimum peak height for a fit to a reference aperture to be accepted. This applies to the peak height in the smoothed image used during the initial search as well as the peak height after profile fitting. It is the check on the smoothed image that is more consistent since the seeing-limited peak height can be highly variable, and the more stringent since smoothing reduces the peak height by seeing**2/(seeing**2+smoothfwhm**2) where ‘seeing’ is the FWHM of the seeing in binned pixels. If smoothfwhm is chosen to be larger than seeing is likely to be, this makes the peak heights more consistent so that the thresholding is better behaved. But, it means you should use a value lower than you might guess from the images. A peak of height h in a smoothed image will contain about (seeing**2+smoothfwhm**2)*h counts in total (but fewer if binned).

hminnrffloat [hidden]

minimum peak height for a fit to a non-reference aperture to be accepted. This applies above all to the smoothed peak height as for hminref.

alphafloat [hidden]

amount by which non-reference apertures are corrected relative to their expected positions when reference apertures are enabled. The idea is that the positions of non-reference relative to reference apertures should vary little, so rather than simply re-positioning independently every frame, one might want to build in a bit of past history. This can be done by setting alpha small. If alpha = 1, then that simply returns to fully independent positioning for each frame.

difffloat [hidden]

maximum difference in the shifts of reference apertures, when more than one has been defined on a CCD. If exceeded, no extraction is performed. This is to guard against perturbations of the reference apertures by meteors, cosmic rays and the like. [unbinned pixels]

rfacfloat [hidden]

target aperture radius relative to the FWHM for ‘variable’ aperture photometry. Usual values 1.5 to 2.5.

rminfloat [hidden]

minimum target aperture radius [unbinned pixels]

rmaxfloat [hidden]

maximum target aperture radius [unbinned pixels]

sinnerfloat [hidden]

inner sky aperture radius [unbinned pixels]

souterfloat [hidden]

outer sky aperture radius [unbinned pixels]

readoutfloat | string [hidden]

readout noise, RMS ADU. Can either be a single value or an hcm file.

gainfloat [hidden]

gain, electrons per ADU. Can either be a single value or an hcm file.

scalefloat [hidden]

image scale in arcsec/pixel

psfgfacfloat [hidden]

multiple of FWHM used to group objects together for PSF fitting

psfwidthint [hidden]

half-width of box used to extract data around objects for PSF fitting

psfpostweakstring [hidden]

During PSF fitting, either hold positions at aperture location (‘fixed’), or fit as part of PSF model (‘variable’)

demask: bool [hidden]

True to attempt to correct for a misplaced frame transfer mask that can affect drift mode data. Sympton is a step in the illumination with Y. This will attempt to subtract the median in X from each window. Do not use unless you need to.

dthresh: float [hidden, if demask]

Threshold (in RMS) to reject pixels in X before taking the median to reduce the effect of bright stars on the median profile. It does this by takeing the average in the Y direction and then rejecting overly bright pixels.

hipercam.scripts.grab(args=None)

grab [source] run [temp] (ndigit) first last trim [twait tmax] ([ncol nrow]) bias [dtype]

This downloads a sequence of images from a raw data file and writes them out to a series CCD / MCCD files.

Parameters:

sourcestring [hidden]

Data source, four options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : ULTRACAM server
‘ul’ : local ULTRACAM .xml/.dat files

The standard start-off default for ``source’’ can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstring

run name to access

tempbool [hidden, defaults to False]

True to indicate that the frames should be written to temporary files with automatically-generated names in the expectation of deleting them later. This also writes out a file listing the names. The aim of this is to allow grab to be used as a feeder for other routines in larger scripts. If temp == True, grab will return with the name of the list of hcm files. Look at ‘makebias’ for an example that uses this feature.

ndigitint [if not temp]

Files created will be written as ‘run005_0013.fits’ etc. ndigit is the number of digits used for the frame number (4 in this case). Any pre-existing files of the same name will be over-written.

firstint

First frame to access

lastint

Last frame to access, 0 for the lot

trimbool

True to trim columns and/or rows off the edges of windows nearest the readout. Particularly useful for ULTRACAM windowed data where the first few rows and columns can contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

twaitfloat [hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [hidden]

maximum time to wait between attempts to find a new exposure, seconds.

biasstring

Name of bias frame to subtract, ‘none’ to ignore.

dtypestring [hidden, defaults to ‘f32’]

Data type on output. Options:

‘f32’ : output as 32-bit floats (default)
‘f64’ : output as 64-bit floats.
‘u16’ : output as 16-bit unsigned integers. A warning will be issued if loss of precision occurs; an exception will be raised if the data are outside the range 0 to 65535.

Note

grab is used by several other scripts such as averun so take great care when changing anything to do with its input parameters.

hipercam.scripts.hfilter(args=None)

hfilter input ccd nx filter [fwhm] output

Filters a multi-CCD image. e.g. smooths it. Can be useful in some analysis steps, e.g. for picking out defects, division by a smoother version of an image can be useful.

Parameters:

inputstring

name of MCCD input file

ccdstring

CCD(s) to filter, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’ or even ‘3 4’ are possible inputs (without the quotes).

filterstring [single character]

type of filter. ‘g’ = gaussian. Uses an FFT-based approach which regards the boundaries as periodic, so you will get significant edge-effects if the values on opposite edges of a window are significantly different from each other.

clobberbool [hidden]

clobber any pre-existing output files

outputstring

name of MCCD output file

hipercam.scripts.hinfo(args=None)

hinfo input

Prints out an hcm file along with its name and the number of CCDs. You will get information CCD-by-CCD first followed by the top level header. You will also get at least partial printouts of the data arrays. If you want still more detail, I recommend the FITS viewer tool ‘fv’, or ‘ds9’ for examining images (also :hplot:).

Parameters:

inputstring

name of the MCCD file

hipercam.scripts.hist(args=None)

hist ccd window x1 x2 nbins gplot (nx) msub (nint) [device width height]

Plots histograms of MCCD objects. Histograms can be powerful diagnostics of CCD problems.

Parameters:

inputstring

name of the MCCD file

ccdstring

CCD or CCDs to plot in histogram form, one CCD per panel. Can be ‘0’ for all of them or a specific set ‘2 3 5’

windowstring

the window label to show. ‘0’ will merge all windows on each CCD.

x1float

lower end of histogram range

x2float

upper end of histoigram range. Set == x1 to get automatic min to max.

nbinsint

number of bins to use for the histogram

gplotbool

plot a gaussian of the same mean and RMS as the data. Note this is not a fit.

nxint

number of panels across to display, prompted if more than one CCD is to be plotted.

msubbool

True/False to subtract median from each window before scaling

nintbool [if msub]

True/False to take the nearest integer of the median or not. Can help with histograms of digitised data.

devicestring [hidden]

Plot device name. Either ‘term’ for an interactive plot or a name like ‘plot.pdf’ for a hard copy.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

hipercam.scripts.hlog2col(args=None)

hlog2col log ap1 ap2

Converts a HiPERCAM ASCII log into one or more column format files of one aperture versus another for each CCD. This tacks on the date of the start of the night in order to distinguish run log names of the the same number.

Parameters:

logstr

name of the log file (should end .log). The output FITS file will have the same root name but end .fits. The routine will abort if there is a pre-existing file of the same name.

originstr

‘h’ or ‘u’ depending upon whether the log file was created with the hipercam or old ultracam pipeline.

offset :float [hidden]

offset in days to subtract from first time to get date at start of the night. 0.7 should mostly work. offset is subtracted and the MJD truncated to integer

ap1str

name of first aperture

ap2str

name of second aperture

typestr

output type ‘l’ for linear; ‘m’ for magnitudes. Linear has the advantage of handling negative values.

The output files will get names like 2021-01-12_run014_2_3_4.dat meaning CCD 2, aperture 3 divided by 4 of run014 from the night starting 2021-01-12.

hipercam.scripts.hlog2fits(args=None)

hlog2fits log

Converts a HiPERCAM ASCII log into a FITS file. As well as a modest reduction in file size (~40%, the ASCII logs are written relatively efficiently), the resulting file is faster to read than the ASCII log so this may be useful for very large log files [test of 78,000 frame file: 12.9 seconds to read the ASCII file, 1.9 to read the FITS version]. The FITS log is also much easier to understand than the ASCII files, but they don’t have all the header information, so are not a replacement. At the moment no significant header information is transferred beyond the CCD names. Each CCD appears as a single binary table, starting at the second HDU (or HDU 1 if you number them 0,1,2 ..). This can be read using hipercam.hlog.Hlog.from_fits().

Parameters:

logstr

name of the log file (should end .log). The output FITS file will have the same root name but end .fits. The routine will abort if there is a pre-existing file of the same name.

originstr

‘h’ or ‘u’ depending upon whether the log file was created with the hipercam or old ultracam pipeline.

NB Because of the danger of over-writing raw data (also ends .fits), this routine will not over-write pre-existing files. You should delete clashing files if you really want to proceed.

hipercam.scripts.hls(args=None)

Gives an ‘ls’-like listing of the runs available on the HiPERCAM file server. Just invoke as ‘hls’ with no arguments. This should work correctly whether it in fact its the HiPERCAM or ULTRACAM file server that is running.

hipercam.scripts.hplot(args=None)

hplot input [device] ccd nx msub [cmap] hsbox iset (ilo ihi | plo phi) xlo xhi ylo yhi [width height]

Plots a multi-CCD image. Can use PGPLOT or matplotlib. The matplotlib version is slightly clunky in its choice of the viewing area but has some features that could be useful, in particular, the interactive plot (device=’/mpl’) allows one to pan and zoom and to compare the same part of multiple CCDs easily.

Parameters:

inputstring

name of MCCD file

devicestring [hidden]

Plot device name. Uses characters after a final trailing ‘/’ to identify the type in PGPLOT style. Thus:

/xs : PGPLOT xserver interactive plot
1/xs : PGPLOT xserver interactive plot called ‘1’
plot.ps/cps : PGPLOT colour postscript called ‘plot.ps’
plot.ps/vps : PGPLOT B&W portrait oriented plot
/mpl : matplotlib interactive plot
plot.pdf/mpl : matplotlib PDF plot
ccdstring

CCD(s) to plot, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’ or even ‘3 4’ are possible inputs (without the quotes). ‘3 4’ will plot CCD ‘3’ and CCD ‘4’. If you want to plot more than one CCD, then you will be prompted for the number of panels in the X direction. This parameter will not be prompted if there is only one CCD in the file.

nxint

number of panels across to display, prompted if more than one CCD is to be plotted.

msubbool

True/False to subtract median from each window before scaling

cmapstr [if matplotlib]

The colour map to use. “Greys” is the usual, but there are many others. Typing an incorrect one will give a list. “none” for matplotlib default.

hsboxint [if device = ‘/mpl’; hidden]

half-width in binned pixels of stats box as offset from central pixel hsbox = 1 gives a 3x3 box; hsbox = 2 gives 5x5 etc.

isetstring [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD bais.

ilofloat [if iset==’d’]

lower intensity level

ihifloat [if iset==’d’]

upper intensity level

plofloat [if iset==’p’]

lower percentile level

phifloat [if iset==’p’]

upper percentile level

xlofloat

left X-limit, for PGPLOT plots. Applies to matplotlib plots to restrict region used to compute percentile limits. This is useful in case where bias strips otherwise distort the plot limits (e.g. ultraspec full frame images)

xhifloat

right X-limit. See comments for xlo as well.

ylofloat

bottom Y-limit. See comments for xlo as well.

yhifloat

top Y-limit. See comments for xlo as well.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

hipercam.scripts.ltimes(args=None)

ltimes [source] run first last [twait tmax tdigit edigit]

Lists timing information from a run in machine-readable space-separated column style. Integer status flags are used, 1=OK, 0=not OK.

For HiPERCAM, the following items are listed: (1) the frame number, (2) the raw GPS timestamp as an MJD, (3) the raw GPS timestamp as a UTC string, (4) a status flag, 0 or 1, then for each CCD: (i) the CCD number [1-5], (ii) the mid exposure time as an MJD, (iii) the mid-exposure time as a UTC string, (iv) the exposure length (seconds), (v) a status flag (0 or 1), determined by the NSKIP parameters for each CCD.

For ULTRACAM, the following items are listed: (1) the frame number, (2) the raw GPS timestamp as an MJD, (3) the raw GPS timestamp as a UTC string, (4) the red & green mid exposure time as an MJD, (5) the red & green mid-exposure time as a UTC string, (6) a status flag to indicate whether the mid-exposure time is considered to be good, (7) the exposure time in seconds, (8) the blue mid-exposure time as an MJD, (9) the blue mid-exposure time as a UTC string, (10) status flag of the blue data, which reflects the “nblue” option where only some of the blue frames are valid data.

For ULTRASPEC, the following items are listed: (1) the frame number, (2) the raw GPS timestamp as an MJD, (3) the raw GPS timestamp as a UTC string, (4) the mid-exposure time as an MJD, (5) the mid-exposure time as a UTC string, (6) a status flag to indicate whether the mid-exposure time is considered to be good, (7) the exposure time in seconds.

Parameters:

sourcestring [hidden]

Data source, two options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : local ULTRA(CAM|SPEC) file
‘ul’ : ULTRA(CAM|SPEC) server
runstring

run number to access, e.g. ‘run0034’

firstint

exposure number to start from. 1 = first frame; set = 0 to always try to get the most recent frame (if it changes)

lastint

last exposure number, 0 for all.

twaitfloat [hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [hidden]

maximum time to wait between attempts to find a new exposure, seconds. Set to 0 to avoid waiting and extra output messages.

tdigitint [hidden]

number of digits of precision for the seconds after the decimal point when reporting the times.

edigitint [hidden]

number of digits of precision after the decimal point when reporting the exposure times

Warning

This routine does not yet work with ULTRACAM data.

hipercam.scripts.ltrans(args=None)

ltrans posns

Reads a file of target positions dumped by ftargets and uses it to derive the transformations needed to co-align images.

Parameters:

posnsstring

file of positions generated by ftargets

cmaxfloat

maximum number of counts at peak allowed for a target. Targets with greater than this will be ignored when computing the transform.

emaxfloat

maximum ratio a/b, major/minor axis elongation. Targets more elongated than this will be ignored.

hipercam.scripts.makebias(args=None)

makebias [source] run first last sigma plot [twait tmax output]

Combines the frames of a single run into a single frame using clipped-mean averaging appropriate for biases. This uses grab to get the frames and combine to combine them. If you already have the frames separately, then just use combine.

Parameters:

sourcestr [hidden]

Data source, four options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : ULTRACAM server
‘ul’ : local ULTRACAM .xml/.dat files

The standard start-off default for ``source’’ can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr

run name to access

firstint

First frame to access

lastint

Last frame to access, 0 for the lot

sigmafloat

The value of ‘sigma’ to pass to the clipped mean combination in ‘combine’

plotbool

make a plot of the mean level versus frame number for each CCD. This can provide a quick check that the frames are not too different. You will need explicitly to close the plot generated at the end of the script

twaitfloat [hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [hidden]

maximum time to wait between attempts to find a new exposure, seconds.

outputstr

name of final combined file. Set by default to match the last part of “run” (but it will have a different extension so they won’t clash)

Note

This routine writes the files returned by ‘grab’ to automatically generated files, typically in /tmp, to avoid polluting the working directory. These are removed at the end, but won’t be if you ctrl-C. You should check your /tmp disk for redundant files in this case.

hipercam.scripts.makedark(args=None)

makedark [source] run first last bias sigma plot [twait tmax output]

Combines the frames of a single run into a single frame using clipped-mean averaging. This uses grab to get the frames and combine to combine them. It subtracts a bias and corrects the exposure time by the exposure time of the bias.

Parameters:

sourcestring [hidden]

Data source, four options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : ULTRACAM server
‘ul’ : local ULTRACAM .xml/.dat files

The standard start-off default for ``source’’ can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstring

run name to access

firstint

First frame to access

lastint

Last frame to access, 0 for the lot

biasstring

Names of bias frame (made e.g. with makebias). This is so that the counts left in the dark frame are genuine dark counts which can be scaled by the ratio of exposure lengths during dark subtraction.

sigmafloat

The value of ‘sigma’ to pass to the clipped mean combination in ‘combine’

twaitfloat [hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [hidden]

maximum time to wait between attempts to find a new exposure, seconds.

outputstring

name of final combined file

Note

This routine writes the files returned by ‘grab’ to automatically generated files, typically in /tmp, to avoid polluting the working directory. These are removed at the end, but won’t be if you ctrl-C. You should check your /tmp disk for redundant files in this case.

hipercam.scripts.makeflat(args=None)

makeflat [source] (run first last [twait tmax] | flist) ngroup bias dark ccd lower upper [clobber] output

Averages a set of images to make a flat field.

Typically flat-fields for HiPERCAM and ULTRA(CAM|SPEC) are taken with a strongly time-variable twilight sky as the Sun sets or rises. A typical flat field run may start out bright, or even saturated, but by the end be only a few thousand counts above bias. Moreover, there are very often stars visible in the images, so we usually take them while offsetting the telescope in a spiral pattern. The challenge is to combine these images while rejecting the stars and saturated frames and giving due weight to the better exposed images. This moreover has to be done for each CCD which vary significantly in sensitivity.

‘makeflat’ does this as follows: given an input list of files (or optionally a single run), it reads them all in, debiases them (optionally), and calculates the mean count level in each CCD, normalises by the mean and writes out the results to temporary files. For each CCD it then sorts the files by their (original) mean level, and for those that lie between defined limits it takes the median of the mean-mormalised frames in groups of defined size. Thus, say one had 75 OK images, then these would be divided into 10 groups, the first 9 having 7 frames, the last having 16. The median average of each of these would be taken. In each case the mean levels would be adjusted to be the same before taking the average to overcome the problem of taking a median of a time-variable sky. The assumption is that while the level may vary, the pattern of the image does not. It is up to the user to check that this is correct. Each of the medians is adjusted to have a mean equal to the sum of the means of the input frames. Finally the normal average of all of these median frames is taken and the mean level of the final output normalised to 1. The first step, taking the median in groups is designed to remove the stars assuming that the telescope was spiralled. The size of the groups (‘ngroup’ below is a crucial parameter in whether this works). A good strategy is to run makeflat for a succession of ever larger ‘ngroup’ and then to divide the results into each other to see if stars are visible.

The final step, the average of the medians with adjusted mean levels, is to ensure that the flats are combined in a way that reflects the level of signal that they have, i.e. to avoid giving equal weights to the median of a series of flats with 20,000 counts per pixel and another series with 1,000 counts per pixel. This somewhat complex procedure is implemented through a series of temporary files which are written and read as the script runs, but deleted at its end. This allows very large numbers to be combined as long as there is enough memory to load ‘ngroup’ CCDs simultaneously, which should usually be fine.

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : ULTRACAM server
‘ul’ : local ULTRACAM .xml/.dat files
‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for ``source’’ can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststr [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame (‘0’ is not supported).

lastint [if source ends ‘s’ or ‘l’]

last exposure number must be >= first or 0 for the whole lot.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

ngroupint

the number of frames. Probably should be at least 5, preferably more. Experiment to see its effect.

biasstring

Name of bias frame to subtract, ‘none’ to ignore.

darkstr

Name of dark frame to subtract, ‘none’ to ignore. Note that it is assumed all CCDs have the same exposure time when making a dark correction.

ccdstr

CCD(s) to process, ‘0’ for all, ‘1 3’ for ‘1’ and ‘3’ only, etc. Would almost always expect this to be set = ‘0’.

lowerlist of floats

Lower limits to the mean count level for a flat to be included. The count level is determined after bias subtraction. Should be the same number as the selected CCDs, and will be assumed to be in the same order. Use this to elminate frames that are of so low a level that the accuracy of the bias subtraction could be a worry. Suggested hipercam values: 3000 for each CCD. Enter values separated by spaces.

upperlist of floats

Upper limits to the mean count level for a flat to be included. The count level is determined after bias subtraction. Should be the same number as the selected CCDs, and will be assumed to be in the same order. Use this to eliminate saturated, peppered or non-linear frames. Suggested hipercam values: 58000, 58000, 58000, 40000 and 40000 for CCDs 1, 2, 3, 4 and 5. Enter values separated by spaces. ULTRACAM values 49000, 29000, 27000 for CCDs 1, 2 and 3.

clobberbool [hidden]

clobber any pre-existing output files

outputstr

output file

hipercam.scripts.mstats(args=None)

mstats [source] run [temp] (ndigit) first last [twait tmax] bias [dtype]

This downloads a sequence of images from a raw data file and writes out stats (min, max, mean, median, rms) for each window to a file

Parameters:

sourcestring [hidden]

Data source, four options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : ULTRACAM server
‘ul’ : local ULTRACAM .xml/.dat files
runstring

run name to access

firstint

First frame to access

lastint

Last frame to access, 0 for the lot

twaitfloat [hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [hidden]

maximum time to wait between attempts to find a new exposure, seconds.

biasstring

Name of bias frame to subtract, ‘none’ to ignore.

formatstring

output format for numbers. e.g. the default ‘9.3f’ might give 12345.678 (9 characters, 3 digits after d.p.)

outfilestring

file for output (extension .stats)

hipercam.scripts.mul(args=None)

mul input1 input2 [ccd win] output

Multiplies two hcm frames and outputs the result. Can be applied only to particular CCDs and windows if wanted.

Parameters:

input1string

first input hcm file

input2string

second input hcm file to multiply into the first.

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as either input1 or input2 in which case the input file will be over-written.

hipercam.scripts.nrtplot(args=None)

nrtplot [source] (run first [twait tmax] | flist) trim ([ncol nrow]) (ccd (nx)) [imwidth pause plotall] bias [lowlevel highlevel] flat defect setup [drurl] [colour] msub iset (ilo ihi | plo phi) xlo xhi ylo yhi profit [method beta fwhm fwhm_min shbox smooth fhbox hmin read gain thresh]

This is ‘nrtplot’ “new” rtplot, a matplotlib-based replacement for the current PGPLOT one. Under development.

Plots a sequence of images as a movie in near ‘real time’, hence ‘rt’. Designed to be used to look at images coming in while at the telescope, ‘rtplot’ comes with many options, a large number of which are hidden by default, and many of which are only prompted if other arguments are set correctly. If you want to see them all, invoke as ‘rtplot prompt’. This is worth doing once to know rtplot’s capabilities.

rtplot can source data from both the ULTRACAM and HiPERCAM servers, from local ‘raw’ ULTRACAM and HiPERCAM files (i.e. .xml + .dat for ULTRACAM, 3D FITS files for HiPERCAM) and from lists of HiPERCAM ‘.hcm’ files.

rtplot optionally allows the selection of targets to be fitted with gaussian or moffat profiles, and, if successful, will plot circles of 2x the measured FWHM in green over the selected targets. This option only works if a single CCD is being plotted.

Parameters:

sourcestring [hidden]

Data source, five options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : ULTRACAM server
‘ul’ : local ULTRACAM .xml/.dat files
‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for ``source’’ can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstring [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststring [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame; set = 0 to always try to get the most recent frame (if it has changed). For data from the HiPERCAM server, a negative number tries to get a frame not quite at the end. i.e. -10 will try to get 10 from the last frame. This is mainly to sidestep a difficult bug with the acquisition system.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

trimbool [if source starts with ‘u’]

True to trim columns and/or rows off the edges of windows nearest the readout which can sometimes contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

ccdstr

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc.

nxint [if more than 1 CCD]

number of panels across to display.

pausefloat [hidden]

seconds to pause between frames (defaults to 0). This is in addition to any time taken per frame to read and display it, so is just to slow things not to set an exact rate.

plotallbool [hidden]

plot all frames regardless of status (i.e. including blank frames when nskips are enabled (defaults to False). The profile fitting will still be disabled for bad frames.

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

lowlevelfloat [hidden]

Level below which a warning about low bias levels is warned. Set=0 to ignore. Applied to first window of first CCD. 2000 about right for ULTRACAM.

highlevelfloat [hidden]

Level above which a warning about high bias levels is warned. Set=0 to ignore. Applied to first window of first CCD. 3500 about right for ULTRACAM.

flatstr

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

defectstr

Name of defect file, ‘none’ to ignore.

setupbool

True/yes to access the current windows from hdriver. Useful during observing when setting up windows, but not normally otherwise. Next argument (hidden) is the URL to get to hdriver. Once setup, you should probably turn this off to avoid overloading hdriver, especially if in drift mode as it makes a request for the windows for every frame.

drurlstr [if setup; hidden]

URL needed to access window setting from the camera driver (ultracam, ultraspec, hipercam). The internal server in the camera driver must be switched on which can be done from the GUI.

cmapstr [hidden]

The matplotlib colour map to use. “Greys” gives the usual greyscale. “none” will give whatever the current default is. Many other choices: “viridis”, “jet”, “hot”, “Oranges” … Enter and invalid one and the program will fail returning a huge list of possibles.

msubbool

subtract the median from each window before scaling for the image display or not. This happens after any bias subtraction.

isetstr [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD basis.

ilofloat [if iset=’d’]

lower intensity level

ihifloat [if iset=’d’]

upper intensity level

plofloat [if iset=’p’]

lower percentile level

phifloat [if iset=’p’]

upper percentile level

xlofloat

left-hand X-limit for plot, initially at least since it is possible to re-size. For iset=’p’ these limits also set the region of the frame over which the percentil will be calculated. You will usually want yhi-ylo ~ xhi-xlo in magnitude because the aspect ratio is preserved.

xhifloat

right-hand X-limit for plot (can be < xlo to invert the display)

ylofloat

lower Y-limit for plot

yhifloat

upper Y-limit for plot (can be < ylo)

profitbool

carry out profile fits or not. If you say yes, then you will get the option of selecting objects with a cursor. The program will then attempt to track these from frame to frame, and fit their profile. You may need to adjust ‘first’ to see anything. The parameters used for profile fits are hidden and you may want to invoke the command with ‘prompt’ the first time you try profile fitting.

nxfint

each fit will be displayed in its own panel in a separate window from the images. This is the maximum number of such panels to use in the X-direction.

methodstr [if profit; hidden]

this defines the profile fitting method, either a gaussian or a moffat profile. The latter is usually best.

betafloat [if profit and method == ‘m’; hidden]

default Moffat exponent

fwhmfloat [if profit; hidden]

default FWHM, unbinned pixels.

fwhm_minfloat [if profit; hidden]

minimum FWHM to allow, unbinned pixels.

shboxfloat [if profit; hidden]

half width of box for searching for a star, unbinned pixels. The brightest target in a region +/- shbox around an intial position will be found. ‘shbox’ should be large enough to allow for likely changes in position from frame to frame, but try to keep it as small as you can to avoid jumping to different targets and to reduce the chances of interference by cosmic rays.

smoothfloat [if profit; hidden]

FWHM for gaussian smoothing, binned pixels. The initial position for fitting is determined by finding the maximum flux in a smoothed version of the image in a box of width +/- shbox around the starter position. Typically should be comparable to the stellar width. Its main purpose is to combat cosmi rays which tend only to occupy a single pixel.

fhboxfloat [if profit; hidden]

half width of box for profile fit, unbinned pixels. The fit box is centred on the position located by the initial search. It should normally be > ~2x the expected FWHM.

hminfloat [if profit; hidden]

height threshold to accept a fit. If the height is below this value, the position will not be updated. This is to help in cloudy conditions.

readfloat [if profit; hidden]

readout noise, RMS ADU, for assigning uncertainties

gainfloat [if profit; hidden]

gain, ADU/count, for assigning uncertainties

threshfloat [if profit; hidden]

sigma rejection threshold for fits

imwidthfloat [hidden]

image display plot width in inches (0 for default)

imheightfloat [hidden]

image display plot height in inches (0 for default)

fwnmaxint [hidden]

maximum number of frames to buffer the FWHM plot (< 2 ==> no plot). This option will only be used if profile fitting is used. It allows the FWHMs of multiple stars to be tracked which might be useful for focussing. The plot starts to “slide” once the buffers are filled.

fwymaxfloat

maximum FWHM to plot on Y-axis [unbinned pixels]

fwwidthfloat [hidden]

FWHM display plot width in inches (0 for default)

fwheightfloat [hidden]

FWHM display plot height in inches (0 for default)

hipercam.scripts.plog(args=None)

plog log [device width height] ccd1 aper1 param1 ccd2 (aper2 param2 scheme) [title]

Provides quick-look plots of HiPERCAM reduce logs.

Parameters:

logstring

name of reduce ASCII log file (text file with loads of columns)

devicestring [hidden, defaults to ‘term’]

‘term’ for interactive plot, file name such as ‘plot.pdf’ for a hardcopy.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

ccd1str

first CCD to consider, e.g. ‘1’

aper1str

first aperture to consider

param1str

first parameter to consider. Choices are ‘x’ = X position, ‘y’ = Y position, ‘f’ = FWHM, ‘b’ = Moffat beta, ‘s’ = sky.

ccd2str

second CCD to consider; ‘!’ to ignore. Can be (and typically would be) the same as ccd1.

aper2str [if ccd2 != ‘!’]

second aperture to consider

param2str [if ccd2 != ‘!’]

second parameter. See param1 for choices

schemestr [if ccd2 != ‘!’]

how to plot if both apertures are chosen. Choices:

‘d’ = difference, i.e. plot 1-2
‘b’ = both plotted on same panel
‘r’ = ratio, i.e. 1 / 2, good for relative photom
‘s’ = scatter plot, 2 vs 1.
titlestr [hidden]

plot title. Defaults to the run number if not specified

Note

Points marked bad, or flagged as having bad times or junk are ignored. i.e. bitmask = BAD_TIME | JUNK is passed to every invocation of Tseries.mplot. Be careful with linked apertures where all x, y, FWHM, beta automatically have negative errors since they are not fitted. See flagcloud for how to flag up junk data.

hipercam.scripts.redanal(args=None)

redanal aper log

This provides some basic stats on a log file such as the fraction of duff points for each aperture of each CCD, how much targets have moved and the like. The aim is to help with setting parameters in the reduce file in difficult cases and to diagnose problems. It will probably be added to with time.

Parameters:

aperstring

the aperture file used for the reduction.

logstring

the log file.

hipercam.scripts.reduce(args=None)

reduce [source] rfile (run first last twait tmax | flist) trim ([ncol nrow]) log lplot implot (ccd nx msub xlo xhi ylo yhi iset (ilo ihi | plo phi))

Reduces a sequence of multi-CCD images, plotting lightcurves as images come in. It can extract with either simple aperture photometry or Tim Naylor’s optimal photometry, on specific targets defined in an aperture file using setaper.

reduce can source data from both the ULTRACAM and HiPERCAM servers, from local ‘raw’ ULTRACAM and HiPERCAM files (i.e. .xml + .dat for ULTRACAM, 3D FITS files for HiPERCAM) and from lists of HiPERCAM ‘.hcm’ files. If you have data from a different instrument you should convert into the FITS-based hcm format.

reduce is primarily configured from a file with extension “.red”. This contains a series of directives, e.g. to say how to re-position and re-size the apertures. An initial reduce file is best generated with the script genred after you have created an aperture file. This contains lots of help on what to do.

A reduce run can be terminated at any point with ctrl-C without doing any harm. You may often want to do this at the start in order to adjust parameters of the reduce file.

Parameters:

sourcestr [hidden]

Data source, five options:

‘hs’: HiPERCAM server
‘hl’: local HiPERCAM FITS file
‘us’: ULTRACAM server
‘ul’: local ULTRACAM .xml/.dat files
‘hf’: list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for ``source’’ can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

runstr [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’ or a file list. If a run, then reduce and log below will be set to have the same name by default.

firstint [if source ends ‘s’ or ‘l’]

first frame to reduce. 1 = first frame; set = 0 to always try to get the most recent frame (if it has changed).

lastint [if source ends ‘s’ or ‘l’, hidden]

last frame to reduce. 0 to just continue until the end. This is not prompted for by default and must be set explicitly. It defaults to 0 if not set. Its purpose is to allow accurate profiling tests.

twaitfloat [if source ends ‘s’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

fliststring [if source ends ‘f’]

name of file list

trimbool

True to trim columns and/or rows off the edges of windows nearest the readout. Particularly useful with ULTRACAM windowed data where the first few rows and columns can contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

rfilestr

the “reduce” file, i.e. ASCII text file suitable for reading by ConfigParser. Best seen by example as it has many parts. If you are reducing a run, this will be set to have the same root name by default (but a different extension to avoid name clashes).

logstr

log file for the results. If you are reducing a run, this will be set to have the same root name by default (but a different extension to avoid name clashes)

tkeepfloat

maximum number of minutes of data to store in internal buffers, 0 for the lot. When large numbers of frames are stored, performance can be slowed (although I am not entirely clear why) in which case it makes sense to lose the earlier points (without affecting the saving to disk). This parameter also gives operation similar to that of “max_xrange” parameter in the ULTRACAM pipeline whereby just the last few minutes are shown.

lplotbool

flag to indicate you want to plot the light curve. Saves time not to especially in high-speed runs.

implotbool

flag to indicate you want to plot images.

ccdstring [if implot]

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc.

nxint [if implot]

number of panels across to display.

msubbool [if implot]

subtract the median from each window before scaling for the image display or not. This happens after any bias subtraction.

xlofloat [if implot]

left-hand X-limit for plot

xhifloat [if implot]

right-hand X-limit for plot (can actually be < xlo)

ylofloat [if implot]

lower Y-limit for plot

yhifloat [if implot]

upper Y-limit for plot (can be < ylo)

isetstring [if implot]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD basis.

ilofloat [if implot and iset=’d’]

lower intensity level

ihifloat [if implot and iset=’d’]

upper intensity level

plofloat [if implot and iset=’p’]

lower percentile level

phifloat [if implot and iset=’p’]

upper percentile level

Warning

The transmission plot generated with reduce is not reliable in the case of optimal photometry since it is highly correlated with the seeing. If you are worried about the transmission during observing, you should always use normal aperture photometry.

hipercam.scripts.rtplot(args=None)

rtplot [source device width height] (run first [twait tmax] | flist) trim ([ncol nrow]) (ccd (nx)) [pause plotall] bias [lowlevel highlevel] flat defect setup [drurl] msub iset (ilo ihi | plo phi) xlo xhi ylo yhi (profit [fdevice fwidth fheight method beta fwhm fwhm_min shbox smooth splot fhbox hmin read gain thresh])

Plots a sequence of images as a movie in near ‘real time’, hence ‘rt’. Designed to be used to look at images coming in while at the telescope, ‘rtplot’ comes with many options, a large number of which are hidden by default, and many of which are only prompted if other arguments are set correctly. If you want to see them all, invoke as ‘rtplot prompt’. This is worth doing once to know rtplot’s capabilities.

rtplot can source data from both the ULTRACAM and HiPERCAM servers, from local ‘raw’ ULTRACAM and HiPERCAM files (i.e. .xml + .dat for ULTRACAM, 3D FITS files for HiPERCAM) and from lists of HiPERCAM ‘.hcm’ files.

rtplot optionally allows the selection of targets to be fitted with gaussian or moffat profiles, and, if successful, will plot circles of 2x the measured FWHM in green over the selected targets. This option only works if a single CCD is being plotted.

Parameters:

sourcestring [hidden]

Data source, five options:

‘hs’ : HiPERCAM server
‘hl’ : local HiPERCAM FITS file
‘us’ : ULTRACAM server
‘ul’ : local ULTRACAM .xml/.dat files
‘hf’ : list of HiPERCAM hcm FITS-format files

‘hf’ is used to look at sets of frames generated by ‘grab’ or converted from foreign data formats. The standard start-off default for ``source’’ can be set using the environment variable HIPERCAM_DEFAULT_SOURCE. e.g. in bash export HIPERCAM_DEFAULT_SOURCE="us" would ensure it always started with the ULTRACAM server by default. If unspecified, it defaults to ‘hl’.

devicestring [hidden]

Plot device. PGPLOT is used so this should be a PGPLOT-style name, e.g. ‘/xs’, ‘1/xs’ etc. At the moment only ones ending /xs are supported.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

runstring [if source ends ‘s’ or ‘l’]

run number to access, e.g. ‘run034’

fliststring [if source ends ‘f’]

name of file list

firstint [if source ends ‘s’ or ‘l’]

exposure number to start from. 1 = first frame; set = 0 to always try to get the most recent frame (if it has changed). For data from the HiPERCAM server, a negative number tries to get a frame not quite at the end. i.e. -10 will try to get 10 from the last frame. This is mainly to sidestep a difficult bug with the acquisition system.

twaitfloat [if source ends ‘s’ or ‘l’; hidden]

time to wait between attempts to find a new exposure, seconds.

tmaxfloat [if source ends ‘s’ or ‘l’; hidden]

maximum time to wait between attempts to find a new exposure, seconds.

trimbool [if source starts with ‘u’]

True to trim columns and/or rows off the edges of windows nearest the readout which can sometimes contain bad data.

ncolint [if trim, hidden]

Number of columns to remove (on left of left-hand window, and right of right-hand windows)

nrowint [if trim, hidden]

Number of rows to remove (bottom of windows)

ccdstring

CCD(s) to plot, ‘0’ for all, ‘1 3’ to plot ‘1’ and ‘3’ only, etc.

nxint [if more than 1 CCD]

number of panels across to display.

pausefloat [hidden]

seconds to pause between frames (defaults to 0)

plotallbool [hidden]

plot all frames regardless of status (i.e. including blank frames when nskips are enabled (defaults to False). The profile fitting will still be disabled for bad frames.

biasstr

Name of bias frame to subtract, ‘none’ to ignore.

lowlevelfloat [hidden]

Level below which a warning about low bias levels is warned. Set=0 to ignore. Applied to first window of first CCD. 2000 about right for ULTRACAM.

highlevelfloat [hidden]

Level above which a warning about high bias levels is warned. Set=0 to ignore. Applied to first window of first CCD. 3500 about right for ULTRACAM.

flatstring

Name of flat field to divide by, ‘none’ to ignore. Should normally only be used in conjunction with a bias, although it does allow you to specify a flat even if you haven’t specified a bias.

defectstring

Name of defect file, ‘none’ to ignore.

setupbool

True/yes to access the current windows from hdriver. Useful during observing when seeting up windows, but not normally otherwise. Next argument (hidden) is the URL to get to hdriver. Once setup, you should probably turn this off to avoid overloading hdriver, especially if in drift mode as it makes a request for the windows for every frame.

drurlstring [if setup; hidden]

URL needed to access window setting from the camera driver (ultracam, ultraspec, hipercam). The internal server in the camera driver must be switched on which can be done from the GUI.

msubbool

subtract the median from each window before scaling for the image display or not. This happens after any bias subtraction.

isetstring [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD basis.

ilofloat [if iset=’d’]

lower intensity level

ihifloat [if iset=’d’]

upper intensity level

plofloat [if iset=’p’]

lower percentile level

phifloat [if iset=’p’]

upper percentile level

xlofloat

left-hand X-limit for plot

xhifloat

right-hand X-limit for plot (can actually be < xlo)

ylofloat

lower Y-limit for plot

yhifloat

upper Y-limit for plot (can be < ylo)

profitbool [if plotting a single CCD only]

carry out profile fits or not. If you say yes, then on the first plot, you will have the option to pick objects with a cursor. The program will then attempt to track these from frame to frame, and fit their profile. You may need to adjust ‘first’ to see anything. The parameters used for profile fits are hidden and you may want to invoke the command with ‘prompt’ the first time you try profile fitting.

fdevicestring [if profit; hidden]

plot device for profile fits, PGPLOT-style name. e.g. ‘/xs’, ‘2/xs’ etc.

fwidthfloat [if profit; hidden]

fit plot width (inches). Set = 0 to let the program choose.

fheightfloat [if profit; hidden]

fit plot height (inches). Set = 0 to let the program choose. BOTH fwidth AND fheight must be non-zero to have any effect

methodstring [if profit; hidden]

this defines the profile fitting method, either a gaussian or a moffat profile. The latter is usually best.

betafloat [if profit and method == ‘m’; hidden]

default Moffat exponent

fwhmfloat [if profit; hidden]

default FWHM, unbinned pixels.

fwhm_minfloat [if profit; hidden]

minimum FWHM to allow, unbinned pixels.

shboxfloat [if profit; hidden]

half width of box for searching for a star, unbinned pixels. The brightest target in a region +/- shbox around an intial position will be found. ‘shbox’ should be large enough to allow for likely changes in position from frame to frame, but try to keep it as small as you can to avoid jumping to different targets and to reduce the chances of interference by cosmic rays.

smoothfloat [if profit; hidden]

FWHM for gaussian smoothing, binned pixels. The initial position for fitting is determined by finding the maximum flux in a smoothed version of the image in a box of width +/- shbox around the starter position. Typically should be comparable to the stellar width. Its main purpose is to combat cosmi rays which tend only to occupy a single pixel.

splotbool [if profit; hidden]

Controls whether an outline of the search box and a target number is plotted (in red) or not.

fhboxfloat [if profit; hidden]

half width of box for profile fit, unbinned pixels. The fit box is centred on the position located by the initial search. It should normally be > ~2x the expected FWHM.

hminfloat [if profit; hidden]

height threshold to accept a fit. If the height is below this value, the position will not be updated. This is to help in cloudy conditions.

readfloat [if profit; hidden]

readout noise, RMS ADU, for assigning uncertainties

gainfloat [if profit; hidden]

gain, ADU/count, for assigning uncertainties

threshfloat [if profit; hidden]

sigma rejection threshold for fits

hipercam.scripts.rupdate(args=None)

rupdate rfile

As changes are made to ‘reduce’, old reduce files can become obsolete. This script tries to bring old reduce files up to date by adding in the new options but in a way that should give the old behaviour. The file is modified in place.

Parameters:

rfilestring

the output reduce file created using setaper. This will be read for the targets. The main target will be assumed to have been called ‘1’, the main comparison ‘2’. If there is a ‘3’ it will be plotted relative to ‘2’; all others will be ignored for plotting purposes.

hipercam.scripts.setaper(args=None)

setaper mccd aper ccd [linput width height] rtarg rsky1 rsky2 xlo xhi ylo yhi nx msub cmap iset (ilo ihi | plo phi) [profit method (beta betafix betamax) fwhm fwfix (fwmin) shbox smooth splot fhbox read gain thresh]

Interactive definition of photometric extraction apertures. This is a matplotlib-based routine allowing you to place apertures on targets using the cursor. It is an essential prerequisite to running reduce.

Parameters:

mccdstring

name of an MCCD file, as produced by e.g. ‘grab’

aperstring

the name of an aperture file. If it exists it will be read so that apertures can be added to it. If it does not exist, it will be created on exiting the routine. The aperture files are is a fairly readable / editiable text format

ccdstring

CCD(s) to plot, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’ or even ‘3 4’ are possible inputs (without the quotes). ‘3 4’ will plot CCD ‘3’ and CCD ‘4’. If you want to plot more than one CCD, then you will be prompted for the number of panels in the X direction. This parameter will not be prompted if there is only one CCD in the file.

linputstring [hidden]

sets the way in which the apertures are labelled. ‘n’ = numerical input, with the program just incrementing by 1 for each successive aperture; ‘s’ = single character (without requiring the user to hit hit <CR>); ‘m’ = multi-character, ending with <CR>. Allowed characters are 0-9, a-z, A-Z, no spaces or punctuation, but a single ‘0’ on its own is not permitted.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

rtargfloat [unbinned pixels]

radius of target aperture. The exact value of this does not matter too much since it is normally overridden in ‘reduce’, but typically one aims for 1.5 to 2.5 x FWHM, seeing, depending upon the target brightness.

rsky1float [unbinned pixels]

inner radius of sky aperture.

rsky2float [unbinned pixels]

radius of target aperture

xlofloat

left X-limit to restrict region used to compute percentile limits. This allows you to only consider useful data if for example there are bias strips, and makes setting the percentile limits less fiddly.

xhifloat

right X-limit. See comments for xlo as well.

ylofloat

bottom Y-limit. See comments for xlo as well.

yhifloat

top Y-limit. See comments for xlo as well.

nxint

number of panels across to display, prompted if more than one CCD is to be plotted.

msubbool

True/False to subtract median from each window before scaling

cmapstr

Colour map to use. “Greys” is usual greyscale. “none” to get matplotlib default.

isetstring [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD bais.

ilofloat [if iset==’d’]

lower intensity level

ihifloat [if iset==’d’]

upper intensity level

plofloat [if iset==’p’]

lower percentile level

phifloat [if iset==’p’]

upper percentile level

profitbool [hidden]

start aperture selection with profile fitting-based position refinement (or not). Can be toggled at any point.

methodstring [hidden]

this defines the profile fitting method, if profile fitting is used to refine the aperture position. Either a gaussian or a moffat profile, ‘g’ or ‘m’. The latter should usually be best.

betafloat [if method == ‘m’; hidden]

default Moffat exponent

betafixbool [if method == ‘m’; hidden]

fix beta or not

beta_maxbool [if method == ‘m’ and not betafix; hidden]

maximum value of beta. Moffat profiles are degenerate with gaussians at large beta so the idea is to avoid wandering to huge beta.

fwhmfloat [hidden]

default FWHM, unbinned pixels. Take care to set it something of the right order of magnitude to avoid spurious rejection of good pixels.

fwfix: bool [hidden]

don’t fit the FWHM. Can be more robust; the position is still fitted.

fwminfloat [if not fwfix; hidden]

minimum FWHM to allow, unbinned pixels.

shboxfloat [hidden]

half width of box for searching for a star, unbinned pixels. The brightest target in a region +/- shbox around an intial position will be found. ‘shbox’ should be large enough to allow for likely changes in position from frame to frame, but try to keep it as small as you can to avoid jumping to different targets and to reduce the chances of interference by cosmic rays.

smoothfloat [hidden]

FWHM for gaussian smoothing, binned pixels. The initial position for fitting is determined by finding the maximum flux in a smoothed version of the image in a box of width +/- shbox around the starter position. Typically should be comparable to the stellar width. Its main purpose is to combat cosmic rays which tend only to occupy a single pixel.

splotbool [hidden]

Controls whether an outline of the search box and a target number is plotted (in red) or not.

fhboxfloat [hidden]

half width of box for profile fit, unbinned pixels. The fit box is centred on the position located by the initial search. It should normally be > ~2x the expected FWHM.

readfloat [hidden]

readout noise, RMS ADU, for assigning uncertainties

gainfloat [hidden]

gain, ADU/count, for assigning uncertainties

threshfloat [hidden]

thresh rejection threshold. Be careful. Bad combinations of this and the initial FWHM can cause wholessale rejection of good pixels, so don’t be overzealous.

ndivint [hidden]

sub-division factor for fits. 0=no sub-division. Otherwise it will used ndiv*ndiv sub-evaluations per unbinned pixel

There are a few conveniences to make setaper easier:

  1. The plot is initialised in pan mode whereby you can move around and scale using the left and right mouse buttons.

  2. All input is accomplished with the keyboard; the mouse buttons are only for navigating the image.

  3. The label input can be switched between sequential numerical, single- and multi-character input (‘linput’).

Various standard keyboard shortcuts (e.g. ‘s’ to save) are disabled as they just confuse things and are of limited use in setaper in any case.

Some aspects of the usage of matplotlib in setaper are tricky. It is possible that particular ‘backends’ will cause problems. I have tested this with Qt4Agg, Qt5agg and GTK3Agg. One aspect is the cursor icon in pan mode is a rather indistinct hand where one can’t tell what is being pointed at. I have therefore suppressed this, but only for the tested backends. Others would need require further investigation.

When in setaper, help is always available by hitting ‘h’. Most of the options are self-evident. A few which may not be are:

e(xtra) : add extra pixels to the target aperture. This allows you
to sculpt the shape of the extraction aperure to perhaps
include the flux of a blended star.

l(ink) : link one aperture to another in the same CCD for
re-positioning. If a target is very faint or will
disappear, this will make sure that its position
is defined relative to another, brighter star. Best
to choose one close by because of potential refractive
distortion.

r(eference): toggle whether an aperture is a reference aperture. The
re-positioning can work in two steps: first position
reference stars, then the other stars, using the reference
stars to provide a first cut at the position shift of the
rest.

C(opy) : copy apertures of the CCD the cursor is in to all others.
This basically clones apertures across the CCDs. You will
need to re-centre each on afterwards.

Warning

The aperture positions immediately after a copy reflect the origin CCD, and may be somewhat off if there are significant offsets between CCDs.

hipercam.scripts.setdefect(args=None)

setdefect mccd defect ccd [linput width height] rtarg rsky1 rsky2 nx msub cmap iset (ilo ihi | plo phi) [profit method beta fwmin fwhm fwfix shbox smooth splot fhbox read gain thresh]

Interactive definition of CCD defects. This is a matplotlib-based routine allowing you to define defects using the cursor.

Parameters:

mccdstring

name of an MCCD file, as produced by e.g. ‘grab’

defectstring

the name of a defect file. If it exists it will be read so that defects can be added to it. If it does not exist, it will be created on exiting the routine. The defect files are in a fairly readable / editiable text format

ccdstring

CCD(s) to plot, ‘0’ for all. If not ‘0’ then ‘1’, ‘2’ or even ‘3 4’ are possible inputs (without the quotes). ‘3 4’ will plot CCD ‘3’ and CCD ‘4’. If you want to plot more than one CCD, then you will be prompted for the number of panels in the X direction. This parameter will not be prompted if there is only one CCD in the file.

widthfloat [hidden]

plot width (inches). Set = 0 to let the program choose.

heightfloat [hidden]

plot height (inches). Set = 0 to let the program choose. BOTH width AND height must be non-zero to have any effect

nxint

number of panels across to display, prompted if more than one CCD is to be plotted.

msubbool

True/False to subtract median from each window before scaling

cmapstr

The colour map to use. “Greys” is the usual; “Greys_r” reverses it. There are many others; typing an incorrect one will give a list. “none” for matplotlib default.

ffieldbool

If True, all defects will be assumed to be flat-field or poor charge transfer defects as opposed to hot pixels. The latter are best set from dark frames, and have a different impact than the first two types in that they are worst for faint targets. Hot pixels and flat-field defects are shown with the same colours for moderate and severe, but different symbols (filled circles for flat-field defects, stars for hot pixels). If you say no to add hot pixels, the line defect option is not available.

hsboxint

half-width in binned pixels of stats box as offset from central pixel hsbox = 1 gives a 3x3 box; hsbox = 2 gives 5x5 etc. This is used by the “show” option when setting defects.

isetstring [single character]

determines how the intensities are determined. There are three options: ‘a’ for automatic simply scales from the minimum to the maximum value found on a per CCD basis. ‘d’ for direct just takes two numbers from the user. ‘p’ for percentile dtermines levels based upon percentiles determined from the entire CCD on a per CCD bais.

ilofloat [if iset==’d’]

lower intensity level

ihifloat [if iset==’d’]

upper intensity level

plofloat [if iset==’p’]

lower percentile level

phifloat [if iset==’p’]

upper percentile level

There are a few conveniences to make setdefect easier:

  1. The plot is initialised in pan mode whereby you can move around and scale using the left and right mouse buttons.

  2. All input is accomplished with the keyboard; the mouse buttons are only for navigating the image.

  3. The label input can be switched between sequential numerical, single- and multi-character input (‘linput’).

Various standard keyboard shortcuts (e.g. ‘s’ to save) are disabled as they just confuse things and are of limited use in setdefect in any case.

Some aspects of the usage of matplotlib in setdefect are tricky. It is possible that particular ‘backends’ will cause problems. I have tested this with Qt4Agg, Qt5agg and GTK3Agg. One aspect is the cursor icon in pan mode is a rather indistinct hand where one can’t tell what is being pointed at. I have therefore suppressed this, but only for the tested backends. Others would need require further investigation.

NB At the end of this routine, it re-orders the defects so that the severe ones follows the moderates. This helps emphasize the severe ones over the moderates when running rtplot.

hipercam.scripts.splice(args=None)

splice input1 input2 ccd [win] output

Merges two multi-CCD images so that parts of input1 are replaced by equivalent parts of input2. This can be useful e.g. to create a combined flat field out of different frames.

Parameters:

input1str

first input frame

input2str

second input frame

ccdstr

the CCD(s) to transfer, ‘0’ for all of them. ‘1 3’, i.e. separate with spaces for more than one CCD.

winstr [hidden]

the window(s) to transfer, ‘0’ for all of them (default). The same windows must exist in the selected CCDs of both inputs

outputstr

output file.

hipercam.scripts.stats(args=None)

stats input [format]

Lists basic stats of a multi-CCD image, i.e. the minimum, maximum, mean, median and standard deviation of each window of each CCD. The output format can be altered to suit preference.

Parameters:

inputstring

name of the MCCD file

formatstring [hidden, defaults to 9.3f]

C-style format code as used in Python format statements for output of the numerical values. e.g. ‘300.00’ is ‘6.2f’ (6 characters toal, 2 after the decimal point), ‘1.22e24’ is ‘.2e’ (as many characters as needed, 2 after the decimal point)

hipercam.scripts.sub(args=None)

sub input1 input2 [ccd win] output

Subtracts two hcm frames and outputs the result. Can be applied only to particular CCDs and windows if wanted.

Parameters:

input1string

first input hcm file

input2string

second input hcm file to subtract from the first.

ccdstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘2 4’ or just one ‘3’

winstring [hidden, defaults to ‘all’]

the CCD or CCDs to apply the operation to. ‘all’ for the whole lot which it returns to by default. Can be several e.g. ‘E2 G1’ or just one ‘H1’. If you specify windows in this manner, it is assumed that all the CCDs chosen in the previous input have the named windows; ‘all’ just applies the operation to all windows regardless.

outputstring

output hcm file name. Can be same as either input1 or input2 in which case the input file will be over-written.

hipercam.scripts.uls(args=None)

Gives an ‘ls’-like listing of the runs available on the ULTRACAM file server. Just invoke as ‘uls’ with no arguments.

See hls for the HiPERCAM equivalent.

1

Several other commands (aligntool, atanalysis, atbytes, digest, hlogger, hmeta, makefield, makedata, pfolder, redplt, tanalysis, tbytes, ulogger) are not documented here as they are of specialist usage. Information on these is however available via pydoc.