10:03:28 - Finished task "Switch HVPA voltage state" 10:03:34 - Calling method FSM.gotoObserving in object FSM 10:03:52 - Checking for high voltage locked pixels... 10:03:53 - Module 85 has 1 pixel(s) locked (HVPA status of 0x3f)! 10:03:53 - Module 94 has 1 pixel(s) locked (HVPA status of 0x7d)! 10:03:53 - Module 97 has 1 pixel(s) locked (HVPA status of 0x5f)! 10:03:53 - Module 101 has 1 pixel(s) locked (HVPA status of 0x7e)! 10:03:53 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:03:54 - Unlocking 5 channel(s)... 10:03:54 - Resetting HVPA channel 6 of drawer 85... 10:03:54 - Resetting HVPA channel 1 of drawer 94... 10:03:54 - Resetting HVPA channel 5 of drawer 97... 10:03:55 - Resetting HVPA channel 0 of drawer 101... 10:03:55 - Resetting HVPA channel 3 of drawer 131... 10:04:22 - Checking for high voltage locked pixels... 10:04:23 - Module 85 has 1 pixel(s) locked (HVPA status of 0x3f)! 10:04:23 - Module 97 has 1 pixel(s) locked (HVPA status of 0x5f)! 10:04:23 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:04:24 - Unlocking 3 channel(s)... 10:04:24 - Resetting HVPA channel 6 of drawer 85... 10:04:24 - Resetting HVPA channel 5 of drawer 97... 10:04:24 - Resetting HVPA channel 3 of drawer 131... 10:04:42 - Stopping NMC 10:04:43 - Found 0 PPS errors. 10:04:44 - Starting NMC 10:04:44 - Clearing PPS error counters 10:04:52 - Checking for high voltage locked pixels... 10:04:53 - Module 85 has 1 pixel(s) locked (HVPA status of 0x3f)! 10:04:53 - Module 97 has 1 pixel(s) locked (HVPA status of 0x5f)! 10:04:53 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:04:54 - Unlocking 3 channel(s)... 10:04:54 - Resetting HVPA channel 6 of drawer 85... 10:04:54 - Resetting HVPA channel 5 of drawer 97... 10:04:54 - Resetting HVPA channel 3 of drawer 131... 10:05:23 - Checking for high voltage locked pixels... 10:05:23 - Module 85 has 1 pixel(s) locked (HVPA status of 0x3f)! 10:05:23 - Module 97 has 1 pixel(s) locked (HVPA status of 0x5f)! 10:05:23 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:05:24 - Unlocking 3 channel(s)... 10:05:24 - Resetting HVPA channel 6 of drawer 85... 10:05:24 - Resetting HVPA channel 5 of drawer 97... 10:05:24 - Resetting HVPA channel 3 of drawer 131... 10:05:53 - Checking for high voltage locked pixels... 10:05:53 - Module 85 has 1 pixel(s) locked (HVPA status of 0x3f)! 10:05:53 - Module 97 has 1 pixel(s) locked (HVPA status of 0x5f)! 10:05:53 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:05:54 - Unlocking 3 channel(s)... 10:05:54 - Resetting HVPA channel 6 of drawer 85... 10:05:54 - Resetting HVPA channel 5 of drawer 97... 10:05:54 - Resetting HVPA channel 3 of drawer 131... 10:06:23 - Checking for high voltage locked pixels... 10:06:23 - Module 97 has 1 pixel(s) locked (HVPA status of 0x5f)! 10:06:23 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:06:24 - Unlocking 2 channel(s)... 10:06:24 - Resetting HVPA channel 5 of drawer 97... 10:06:24 - Resetting HVPA channel 3 of drawer 131... 10:06:53 - Checking for high voltage locked pixels... 10:06:53 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:06:54 - Unlocking 1 channel(s)... 10:06:54 - Resetting HVPA channel 3 of drawer 131... 10:07:23 - Checking for high voltage locked pixels... 10:07:23 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:07:24 - Unlocking 1 channel(s)... 10:07:24 - Resetting HVPA channel 3 of drawer 131... 10:07:53 - Checking for high voltage locked pixels... 10:07:53 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:07:54 - Unlocking 1 channel(s)... 10:07:54 - Resetting HVPA channel 3 of drawer 131... 10:08:23 - Checking for high voltage locked pixels... 10:08:23 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:08:24 - Unlocking 1 channel(s)... 10:08:24 - Resetting HVPA channel 3 of drawer 131... 10:08:53 - Checking for high voltage locked pixels... 10:08:53 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:08:54 - Unlocking 1 channel(s)... 10:08:54 - Resetting HVPA channel 3 of drawer 131... 10:09:23 - Checking for high voltage locked pixels... 10:09:23 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:09:24 - Unlocking 1 channel(s)... 10:09:24 - Resetting HVPA channel 3 of drawer 131... 10:09:53 - Checking for high voltage locked pixels... 10:09:53 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:09:54 - Unlocking 1 channel(s)... 10:09:54 - Resetting HVPA channel 3 of drawer 131... 10:10:23 - Checking for high voltage locked pixels... 10:10:23 - Module 131 has 1 pixel(s) locked (HVPA status of 0x77)! 10:10:52 - Unlocking 1 channel(s)... 10:10:52 - Resetting HVPA channel 3 of drawer 131... 10:10:55 - Checking for high voltage locked pixels... 10:11:23 - Checking for high voltage locked pixels... 10:11:51 - Starting task "Check high voltages" 10:11:52 - Found 53 drawer(s) in Nominal state, 0 in Low state and 0 in Wait state 10:11:52 - Checking for high voltage locked pixels... 10:12:00 - Actual voltage is between 945.50 V and 1167.54 V. 10:12:00 - Low voltages range between 300.00 V and 300.00 V. 10:12:09 - Stopping NMC 10:12:10 - Found 0 PPS errors. 10:12:10 - Reference voltages range between 945.26 V and 1167.15 V. 10:12:11 - Starting NMC 10:12:11 - Clearing PPS error counters 10:12:19 - Maximum difference between expected and actual voltages is 0.85 V 10:12:22 - Checking for high voltage locked pixels... 10:12:23 - Maximum of current limits is 5647.07 uA 10:12:23 - Checking status of HVPA module in 53 drawer(s)... 10:12:24 - Checking status of HVPA channels in 53 drawer(s)... 10:12:28 - HV is stabilized in all channels. 10:12:28 - Checking reference and low voltages against /local/home/cta_test_bench/config/2020/20200610/config_tower_53modules.NMC.xml 10:12:33 - Done 10:12:33 - Finished task "Check high voltages" 10:12:36 - Checking UCTS clock... 10:12:37 - UCTS and control PC clocks differ by 4.2 seconds 10:12:37 - Configuring UCTS 10:12:37 - Configuring TIB... 10:12:37 - Setting TIB forced mode to False 10:12:37 - Writing False to Camera.TIB.Forced.ForcedMode.ForcedMode_v 10:12:37 - Status: TibState.PpsStopped 10:12:37 - Alarms: AlarmVector.ReadError 10:12:37 - Connection status: False TXclient_alive: False 10:12:37 - Entering set_external_delay 10:12:37 - Setting ExternalDelayLocal to 0.000 ns 10:12:37 - Writing 0.0 to Camera.TIB.ExternalDelay.ExternalDelayLocal.ExternalDelayLocal_v 10:12:37 - Entering set_external_delay 10:12:37 - Setting ExternalDelayCalibration to 65.000 ns 10:12:37 - Writing 65.0 to Camera.TIB.ExternalDelay.ExternalDelayCalibration.ExternalDelayCalibration_v 10:12:37 - Entering set_external_delay 10:12:37 - Setting ExternalDelaySphe to 150.000 ns 10:12:37 - Writing 150.0 to Camera.TIB.ExternalDelay.ExternalDelaySphe.ExternalDelaySphe_v 10:12:37 - Setting TIB trigger mode to 0x75 = TriggerMask.SlowControl|Pedestal|Software|Calibration|Mono 10:12:37 - Writing 117 to Camera.TIB.TriggerMode.ModeMask.ModeMask_v 10:12:37 - Writing 1 to Camera.TIB.InputsEnable.InputMask.InputMask_v 10:12:37 - Writing False to Camera.TIB.BUSY.DisableBusy.DisableBusy_v 10:12:37 - Writing 0.0 to Camera.TIB.BUSY.ExtraDelayBusy.ExtraDelayBusy_v 10:12:37 - Writing False to Camera.TIB.Throttler.DisableThrottler.DisableThrottler_v 10:12:37 - Writing 255 to Camera.TIB.Throttler.MaxTriggers50us.MaxTriggers50us_v 10:12:37 - Writing 255 to Camera.TIB.Throttler.MaxTriggers1ms.MaxTriggers1ms_v 10:12:37 - Status: TibState.PpsStopped 10:12:37 - Alarms: AlarmVector.ReadError 10:12:37 - Connection status: False TXclient_alive: False 10:12:37 - Writing True to Camera.TIB.Pedestal.PeriodicPedestalEnable.PeriodicPedestalEnable_v 10:12:37 - Writing 10 to Camera.TIB.Pedestal.PeriodicPedestalFrequency.PeriodicPedestalFrequency_v 10:12:37 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v 10:12:37 - Writing False to Camera.TIB.Pedestal.DigitalPedestalEnable.DigitalPedestalEnable_v 10:12:37 - Writing 10 to Camera.TIB.Pedestal.DigitalPedestal_config_Rate.DigitalPedestal_config_Rate_v 10:12:37 - Writing 375764 to Camera.TIB.Pedestal.DigitalPedestalSeed.DigitalPedestalSeed_v 10:12:37 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v 10:12:37 - Writing False to Camera.TIB.Pedestal.AnalogPedestalEnable.AnalogPedestalEnable_v 10:12:37 - Writing 3000.0 to Camera.TIB.Pedestal.AnalogPedestalThreshold.AnalogPedestalThreshold_v 10:12:37 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v 10:12:40 - Connecting TIB to camera server at 10.10.4.12:222 10:12:40 - Starting TIB... 10:12:40 - Status: TibState.PpsStopped 10:12:40 - Alarms: AlarmVector.ReadError 10:12:40 - Connection status: False TXclient_alive: False 10:12:40 - Resetting run 10:12:40 - Wait for UCTS external trigger before enabling trigger... 10:12:40 - Status: TibState.Standby 10:12:40 - Alarms: AlarmVector.NoAlarm 10:12:40 - Connection status: False TXclient_alive: True 10:12:40 - Status: TibState.Standby 10:12:40 - Alarms: AlarmVector.NoAlarm 10:12:40 - Connection status: False TXclient_alive: True 10:12:41 - Status: TibState.Armed 10:12:41 - Alarms: AlarmVector.NoAlarm 10:12:41 - Connection status: True TXclient_alive: True 10:12:43 - Configuring calibration box... 10:12:43 - Configuring calibration box with leds=0x40 voltage=10.00 frequency=300.0/30 central_current=0.0 duration=0 width=1 lemo_out=True fiber1_out=True fiber2_out=True lemo_in=False enabled=False 10:12:45 - Starting flashes... 10:12:49 - Starting TIB triggers... 10:12:49 - Enabling trigger 10:12:49 - Status: TibState.Running 10:12:49 - Alarms: AlarmVector.NoAlarm 10:12:49 - Connection status: True TXclient_alive: True 10:12:52 - Checking for high voltage locked pixels... 10:12:56 - Status: TibState.Running ConnectionStatus: True TXclient_alive: True 10:12:56 - Alarms: AlarmVector.ReadError 10:12:56 - Temperature: 33.56 10:12:56 - Local camera rate: 11.0 Hz 10:12:56 - Stereo trigger rate: 0.0 Hz 10:12:56 - SPE trigger rate: 0.0 Hz 10:12:56 - Calibration trigger rate: 10.0 Hz 10:12:56 - Software trigger rate: 0.0 Hz 10:12:56 - Pedestal trigger rate: 10.0 Hz 10:12:56 - Collected trigger rate: 21.0 Hz 10:12:56 - Busy rate: 0.0 Hz 10:12:56 - Camera trigger rate: 21.0 Hz 10:13:22 - Checking for high voltage locked pixels... 10:13:53 - Checking for high voltage locked pixels... 10:14:23 - Checking for high voltage locked pixels... 10:14:53 - Checking for high voltage locked pixels... 10:15:23 - Checking for high voltage locked pixels... 10:15:53 - Checking for high voltage locked pixels... 10:16:23 - Checking for high voltage locked pixels... 10:16:53 - Checking for high voltage locked pixels... 10:17:23 - Checking for high voltage locked pixels... 10:17:52 - Checking for high voltage locked pixels... 10:18:23 - Checking for high voltage locked pixels... 10:18:53 - Checking for high voltage locked pixels... 10:19:23 - Checking for high voltage locked pixels... 10:19:52 - Checking for high voltage locked pixels... 10:20:22 - Checking for high voltage locked pixels... 10:20:52 - Checking for high voltage locked pixels... 10:21:23 - Checking for high voltage locked pixels... 10:21:53 - Checking for high voltage locked pixels... 10:22:09 - Status: TibState.Running ConnectionStatus: True TXclient_alive: True 10:22:09 - Alarms: AlarmVector.ReadError 10:22:09 - Temperature: 34.19 10:22:09 - Local camera rate: 11.0 Hz 10:22:09 - Stereo trigger rate: 0.0 Hz 10:22:09 - SPE trigger rate: 0.0 Hz 10:22:09 - Calibration trigger rate: 10.0 Hz 10:22:09 - Software trigger rate: 0.0 Hz 10:22:09 - Pedestal trigger rate: 10.0 Hz 10:22:09 - Collected trigger rate: 21.0 Hz 10:22:09 - Busy rate: 0.0 Hz 10:22:09 - Camera trigger rate: 21.0 Hz 10:22:23 - Checking for high voltage locked pixels... 10:22:53 - Checking for high voltage locked pixels... 10:23:23 - Checking for high voltage locked pixels... 10:23:53 - Checking for high voltage locked pixels... 10:24:23 - Checking for high voltage locked pixels... 10:24:52 - Checking for high voltage locked pixels... 10:25:23 - Checking for high voltage locked pixels... 10:25:52 - Checking for high voltage locked pixels... 10:26:23 - Checking for high voltage locked pixels... 10:26:53 - Checking for high voltage locked pixels... 10:27:23 - Checking for high voltage locked pixels... 10:27:53 - Checking for high voltage locked pixels... 10:28:23 - Checking for high voltage locked pixels... 10:28:52 - Checking for high voltage locked pixels... 10:29:22 - Checking for high voltage locked pixels... 10:29:52 - Checking for high voltage locked pixels... 10:30:22 - Checking for high voltage locked pixels... 10:30:53 - Checking for high voltage locked pixels... 10:31:23 - Checking for high voltage locked pixels... 10:31:53 - Checking for high voltage locked pixels... 10:32:23 - Checking for high voltage locked pixels... 10:32:50 - Stopping TIB... 10:32:50 - Status: TibState.Running 10:32:50 - Alarms: AlarmVector.ReadError 10:32:50 - Connection status: True TXclient_alive: True 10:32:50 - Disabling trigger 10:32:50 - Disabling PPS 10:32:51 - Status: TibState.PpsStopped 10:32:51 - Alarms: AlarmVector.ReadError 10:32:51 - Connection status: True TXclient_alive: True 10:32:51 - Closing TIB connection to camera server 10:32:52 - Checking for high voltage locked pixels... 10:32:54 - Status: TibState.PpsStopped 10:32:54 - Alarms: AlarmVector.ReadError 10:32:54 - Connection status: False TXclient_alive: False 10:32:54 - Run duration: 0:20:09 10:32:54 - Local camera triggers: 13973 10:32:54 - Slow control triggers: 0 10:32:54 - Single photo-electron triggers: 0 10:32:54 - Calibration triggers: 12055 10:32:54 - Software triggers from UCTS: 0 10:32:54 - Pedestal triggers: 12085 10:32:54 - Collected input triggers: 25931 10:32:54 - Busy triggers: 2 10:32:54 - Camera event triggers sent to clusters: 25929 10:33:08 - Calling method FSM.gotoReady in object FSM 10:33:12 - Stopping NMC 10:33:13 - Found 0 PPS errors. 10:33:15 - Starting task "Switch HVPA voltage state" 10:33:15 - Setting HVPA current limit to 12489.75 uA 10:33:22 - Checking for high voltage locked pixels... 10:33:33 - Setting HVPA ramp speed to 500 V/s 10:33:35 - Switching HVPA to state Wait 10:33:44 - Setting HVPA current limits... 10:33:52 - Checking for high voltage locked pixels...