16:39:40 - NMC is in state NmcState.Configured and knows of 52 drawer(s)
16:39:41 - Found 0 PPS errors.
16:39:50 - Starting task "Switch HVPA voltage state"
16:39:50 - Setting HVPA current limit to 12489.75 uA
16:40:03 - Calling method FSM.gotoReady in object FSM
16:40:07 - Setting HVPA ramp speed to 25 V/s
16:40:09 - Switching HVPA to state Nominal
16:40:18 - Setting HVPA current limits...
16:40:40 - Finished task "Switch HVPA voltage state"
16:40:45 - High voltage locked channels will be checked every 30 s
16:40:46 - Starting task "Check high voltages"
16:40:47 - Found 52 drawer(s) in Nominal state, 0 in Low state and 0 in Wait state
16:40:55 - Actual voltage is between 3.05 V and 1148.02 V.
16:40:56 - Low voltages range between 300.00 V and 300.00 V. 
16:41:05 - Reference voltages range between 945.26 V and 1167.15 V. 
16:41:14 - Maximum difference between expected and actual voltages is 1060.53 V
16:41:15 - Checking for high voltage locked pixels...
16:41:16 - Module 81 has 1 pixel(s) locked (HVPA status of 0x3f)!
16:41:16 - Module 146 has 1 pixel(s) locked (HVPA status of 0x7d)!
16:41:17 - Module 163 has 1 pixel(s) locked (HVPA status of 0x77)!
16:41:17 - Unlocking 3 channel(s)...
16:41:17 - Resetting HVPA channel 6 of drawer 81...
16:41:17 - Resetting HVPA channel 1 of drawer 146...
16:41:18 - Resetting HVPA channel 3 of drawer 163...
16:41:19 - Maximum of current limits is 12489.75 uA
16:41:19 - Checking status of HVPA module in 52 drawer(s)...
16:41:19 - Checking status of HVPA channels in 52 drawer(s)...
16:41:22 - HVPA channel 3 of drawer 163 has status 0x8003!
16:41:23 - HVPA channel 6 of drawer 81 has status 0x8003!
16:41:23 - HV is locked in 2 channel(s), was in rise default in 0 channel(s) and was still in progress in 1 channel(s)
16:41:23 - Resetting 2 channel(s)...
16:41:23 - Resetting HVPA channel 3 of drawer 163...
16:41:24 - Resetting HVPA channel 6 of drawer 81...
16:41:35 - Checking reference and low voltages against /local/home/cta_test_bench/config/2020/20200128/config_tower_52modules.NMC.xml
16:41:38 - Reference voltage of channel 1 in drawer #146: expected 1064.19, read 0.00
16:41:40 - Done
16:41:40 - Finished task "Check high voltages"
16:41:42 - Calling method FSM.gotoObserving in object FSM
16:41:45 - Checking for high voltage locked pixels...
16:41:46 - Module 146 has 1 pixel(s) locked (HVPA status of 0x7d)!
16:41:47 - Unlocking 1 channel(s)...
16:41:47 - Resetting HVPA channel 1 of drawer 146...
16:42:15 - Checking for high voltage locked pixels...
16:42:45 - Checking for high voltage locked pixels...
16:43:15 - Checking for high voltage locked pixels...
16:43:23 - Starting NMC
16:43:23 - Clearing PPS error counters
16:43:28 - Starting task "Power off channels not corresponding to a configured module"
16:43:28 - Powering off all but 52 DTC channel(s)
16:43:31 - Executing command '/opt/cta/bin/cta_ctrl --slot 1 --channel 4 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:32 - Executing command '/opt/cta/bin/cta_ctrl --slot 1 --channel 11 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:34 - Executing command '/opt/cta/bin/cta_ctrl --slot 1 --channel 15 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:36 - Executing command '/opt/cta/bin/cta_ctrl --slot 5 --channel 3 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:38 - Executing command '/opt/cta/bin/cta_ctrl --slot 5 --channel 9 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:39 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 1 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:41 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 2 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:43 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 4 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:45 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 6 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:45 - Checking for high voltage locked pixels...
16:43:46 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 7 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:48 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 8 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:50 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 9 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:52 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 10 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:53 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 11 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:55 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 12 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:57 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 13 --power_off' on Digital Trigger Crate at 10.10.4.57
16:43:59 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 14 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:00 - Executing command '/opt/cta/bin/cta_ctrl --slot 6 --channel 15 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:02 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 1 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:04 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 2 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:05 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 3 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:07 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 4 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:09 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 5 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:11 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 6 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:12 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 7 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:14 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 8 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:15 - Checking for high voltage locked pixels...
16:44:16 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 9 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:18 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 10 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:19 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 11 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:21 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 12 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:23 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 13 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:25 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 14 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:26 - Executing command '/opt/cta/bin/cta_ctrl --slot 9 --channel 15 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:28 - Executing command '/opt/cta/bin/cta_ctrl --slot 13 --channel 1 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:30 - Executing command '/opt/cta/bin/cta_ctrl --slot 13 --channel 2 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:32 - Executing command '/opt/cta/bin/cta_ctrl --slot 13 --channel 6 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:33 - Executing command '/opt/cta/bin/cta_ctrl --slot 13 --channel 13 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:35 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 1 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:37 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 2 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:39 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 3 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:40 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 5 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:42 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 8 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:44 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 9 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:45 - Checking for high voltage locked pixels...
16:44:46 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 10 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:47 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 11 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:49 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 12 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:51 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 13 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:53 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 14 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:54 - Executing command '/opt/cta/bin/cta_ctrl --slot 14 --channel 15 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:56 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 1 --power_off' on Digital Trigger Crate at 10.10.4.57
16:44:58 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 2 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:00 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 3 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:01 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 4 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:03 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 5 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:05 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 6 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:07 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 7 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:08 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 8 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:10 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 11 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:12 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 12 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:14 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 13 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:15 - Checking for high voltage locked pixels...
16:45:15 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 14 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:17 - Executing command '/opt/cta/bin/cta_ctrl --slot 15 --channel 15 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:19 - Executing command '/opt/cta/bin/cta_ctrl --slot 19 --channel 3 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:21 - Executing command '/opt/cta/bin/cta_ctrl --slot 19 --channel 4 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:22 - Executing command '/opt/cta/bin/cta_ctrl --slot 19 --channel 5 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:24 - Executing command '/opt/cta/bin/cta_ctrl --slot 19 --channel 6 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:25 - NMC is in state NmcState.Running and knows of 52 drawer(s)
16:45:25 - Found 0 PPS errors.
16:45:26 - Executing command '/opt/cta/bin/cta_ctrl --slot 19 --channel 11 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:28 - Executing command '/opt/cta/bin/cta_ctrl --slot 19 --channel 14 --power_off' on Digital Trigger Crate at 10.10.4.57
16:45:30 - Finished task "Power off channels not corresponding to a configured module"
16:45:33 - Starting task "Set DTC trigger masks"
16:45:34 - Configuring DTC trigger masks for 52 configured module(s)
16:45:34 - Setting DTC slot 1 trigger mask to 0x77ee
16:45:37 - Setting DTC slot 5 trigger mask to 0xfdf6
16:45:41 - Setting DTC slot 6 trigger mask to 0x0028
16:45:44 - Setting DTC slot 9 trigger mask to 0x0000
16:45:45 - Checking for high voltage locked pixels...
16:45:48 - Setting DTC slot 13 trigger mask to 0xdfb8
16:45:51 - Setting DTC slot 14 trigger mask to 0x00d0
16:45:55 - Setting DTC slot 15 trigger mask to 0x0600
16:45:58 - Setting DTC slot 19 trigger mask to 0xb786
16:46:02 - Finished task "Set DTC trigger masks"
16:46:15 - Checking for high voltage locked pixels...
16:46:21 - Starting task "Configure modules"
16:46:21 - Setting L0 thresholds to 7.0 p.e. per pixel using L0 slopes and offsets in /local/home/cta_test_bench/data/2019/20191203/l0_threshold_slope_and_offset_feb.h5
16:46:31 - Configuring modules with configuration /local/home/cta_test_bench/config/2020/20200128/config_2020-01-28_16-46-21.NMC.xml
16:46:31 - Switching off high voltages in configuration
16:46:31 - Removing neighbouring module registers
16:46:35 - Successfully configured 52 module(s)
16:46:40 - NMC is in state NmcState.Configured and knows of 52 drawer(s)
16:46:41 - Found 0 PPS errors.
16:46:41 - Finished task "Configure modules"
16:46:43 - Starting task "Switch HVPA voltage state"
16:46:43 - Setting HVPA current limit to 12489.75 uA
16:46:45 - Checking for high voltage locked pixels...
16:47:00 - Setting HVPA ramp speed to 25 V/s
16:47:02 - Switching HVPA to state Nominal
16:47:11 - Setting HVPA current limits...
16:47:15 - Checking for high voltage locked pixels...
16:47:33 - Finished task "Switch HVPA voltage state"
16:47:45 - Checking for high voltage locked pixels...
16:48:15 - Checking for high voltage locked pixels...
16:48:45 - Checking for high voltage locked pixels...
16:48:50 - Starting NMC
16:48:50 - Clearing PPS error counters
16:48:56 - Configuring UCTS
16:48:56 - Configuring TIB...
16:48:56 - Setting TIB forced mode to False
16:48:56 - Writing False to Camera.TIB.Forced.ForcedMode.ForcedMode_v
16:48:56 - Status: TibState.RunStopped
16:48:56 - Alarms: AlarmVector.NoAlarm
16:48:56 - Connection status: False
16:48:56 - Disabling PPS
16:48:58 - Status: TibState.PpsStopped
16:48:58 - Alarms: AlarmVector.NoAlarm
16:48:58 - Connection status: False
16:48:58 - Entering set_external_delay
16:48:58 - Setting ExternalDelayLocal to 0.000 ns
16:48:58 - Writing 0.0 to Camera.TIB.ExternalDelay.ExternalDelayLocal.ExternalDelayLocal_v
16:48:58 - Entering set_external_delay
16:48:58 - Setting ExternalDelayCalibration to 98.000 ns
16:48:58 - Writing 98.0 to Camera.TIB.ExternalDelay.ExternalDelayCalibration.ExternalDelayCalibration_v
16:48:58 - Entering set_external_delay
16:48:58 - Setting ExternalDelaySphe to 150.000 ns
16:48:58 - Writing 150.0 to Camera.TIB.ExternalDelay.ExternalDelaySphe.ExternalDelaySphe_v
16:48:59 - Setting TIB trigger mode to 0x51 = TriggerMask.SlowControl|Software|Mono
16:48:59 - Writing 81 to Camera.TIB.TriggerMode.ModeMask.ModeMask_v
16:48:59 - Writing 0 to Camera.TIB.InputsEnable.InputMask.InputMask_v
16:48:59 - Writing False to Camera.TIB.BUSY.DisableBusy.DisableBusy_v
16:48:59 - Writing 0.0 to Camera.TIB.BUSY.ExtraDelayBusy.ExtraDelayBusy_v
16:48:59 - Writing False to Camera.TIB.Throttler.DisableThrottler.DisableThrottler_v
16:48:59 - Writing 255 to Camera.TIB.Throttler.MaxTriggers50us.MaxTriggers50us_v
16:48:59 - Writing 255 to Camera.TIB.Throttler.MaxTriggers1ms.MaxTriggers1ms_v
16:48:59 - Status: TibState.PpsStopped
16:48:59 - Alarms: AlarmVector.NoAlarm
16:48:59 - Connection status: False
16:48:59 - Writing False to Camera.TIB.Pedestal.PeriodicPedestalEnable.PeriodicPedestalEnable_v
16:48:59 - Writing 10 to Camera.TIB.Pedestal.PeriodicPedestalFrequency.PeriodicPedestalFrequency_v
16:48:59 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
16:48:59 - Writing False to Camera.TIB.Pedestal.DigitalPedestalEnable.DigitalPedestalEnable_v
16:48:59 - Writing 10 to Camera.TIB.Pedestal.DigitalPedestal_config_Rate.DigitalPedestal_config_Rate_v
16:48:59 - Writing 375764 to Camera.TIB.Pedestal.DigitalPedestalSeed.DigitalPedestalSeed_v
16:48:59 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
16:48:59 - Writing False to Camera.TIB.Pedestal.AnalogPedestalEnable.AnalogPedestalEnable_v
16:48:59 - Writing 3000.0 to Camera.TIB.Pedestal.AnalogPedestalThreshold.AnalogPedestalThreshold_v
16:48:59 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
16:49:00 - Connecting TIB to camera server at 10.10.4.12:222
16:49:00 - Starting TIB...
16:49:00 - Status: TibState.PpsStopped
16:49:00 - Alarms: AlarmVector.NoAlarm
16:49:00 - Connection status: False
16:49:00 - Resetting run
16:49:00 - Wait for UCTS external trigger before enabling trigger...
16:49:00 - Status: TibState.Standby
16:49:00 - Alarms: AlarmVector.NoAlarm
16:49:00 - Connection status: False
16:49:00 - Status: TibState.Standby
16:49:00 - Alarms: AlarmVector.NoAlarm
16:49:00 - Connection status: False
16:49:01 - Status: TibState.Armed
16:49:01 - Alarms: AlarmVector.NoAlarm
16:49:01 - Connection status: True
16:49:03 - Starting TIB triggers...
16:49:03 - Enabling trigger
16:49:03 - Status: TibState.Running
16:49:03 - Alarms: AlarmVector.NoAlarm
16:49:03 - Connection status: True
16:49:09 - Configuring calibration box...
16:49:09 - Configuring calibration box with leds=0x40 voltage=10.00 frequency=1000.0/1 central_current=0.0 duration=100 width=1 lemo_out=False fiber1_out=False fiber2_out=False lemo_in=False enabled=False
16:49:12 - Starting flashes...
16:49:15 - Checking for high voltage locked pixels...
16:49:30 - Stopping TIB...
16:49:30 - Status: TibState.Running
16:49:30 - Alarms: AlarmVector.ReadError
16:49:30 - Connection status: True
16:49:30 - Disabling trigger
16:49:30 - Disabling PPS
16:49:31 - Status: TibState.PpsStopped
16:49:31 - Alarms: AlarmVector.ReadError
16:49:31 - Connection status: True
16:49:31 - Closing TIB connection to camera server
16:49:32 - Status: TibState.PpsStopped
16:49:32 - Alarms: AlarmVector.ReadError
16:49:32 - Connection status: False
16:49:32 - Run duration: 0:00:29
16:49:32 - Local camera triggers: 0
16:49:32 - Slow control triggers: 0
16:49:32 - Single photo-electron triggers: 0
16:49:32 - Calibration triggers: 0
16:49:32 - Software triggers from UCTS: 0
16:49:32 - Pedestal triggers: 0
16:49:32 - Collected input triggers: 9989
16:49:32 - Busy triggers: 0
16:49:32 - Camera event triggers sent to clusters: 9989
16:49:34 - Calling method FSM.gotoReady in object FSM