13:55:41 - High voltage locked channels will be checked every 30 s
13:56:11 - Checking for high voltage locked pixels...
13:56:13 - Module 86 has 1 pixel(s) locked (HVPA status of 0x7b)!
13:56:13 - Module 91 has 1 pixel(s) locked (HVPA status of 0x3f)!
13:56:16 - Module 248 has 1 pixel(s) locked (HVPA status of 0x77)!
13:56:25 - HVPA channel 2 of drawer 86 has status 0x8003!
13:56:25 - HVPA channel 2 in module 86 is locked!
13:56:28 - HVPA channel 3 of drawer 248 has status 0x8003!
13:56:28 - HVPA channel 3 in module 248 is locked!
13:56:41 - HVPA channel 6 of drawer 91 has status 0x8003!
13:56:41 - HVPA channel 6 in module 91 is locked!
13:56:41 - HV is locked in 3 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
13:56:41 - Unlocking 3 channel(s)...
13:56:41 - ("NmcError: Failed to write 'NectarFrontEnd_086.Hvpa.Hvpa2.Disable': status Bad statuses are present, code 0xfffffffb, status NmcStatus.Mismatch", 4294967291)
13:56:42 - Checking for high voltage locked pixels...
13:56:44 - Module 86 has 1 pixel(s) locked (HVPA status of 0x7b)!
13:56:44 - Module 91 has 1 pixel(s) locked (HVPA status of 0x3f)!
13:56:47 - Module 248 has 1 pixel(s) locked (HVPA status of 0x77)!
13:56:56 - HVPA channel 2 of drawer 86 has status 0x8003!
13:56:56 - HVPA channel 2 in module 86 is locked!
13:57:00 - HVPA channel 3 of drawer 248 has status 0x8003!
13:57:00 - HVPA channel 3 in module 248 is locked!
13:57:12 - HVPA channel 6 of drawer 91 has status 0x8003!
13:57:12 - HVPA channel 6 in module 91 is locked!
13:57:12 - HV is locked in 3 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
13:57:12 - Unlocking 3 channel(s)...
13:57:14 - Checking for high voltage locked pixels...
13:57:19 - Module 248 has 1 pixel(s) locked (HVPA status of 0x77)!
13:57:29 - HVPA channel 2 of drawer 86 has status 0x0003!
13:57:29 - HVPA channel 2 of drawer 228 has status 0x0003!
13:57:32 - HVPA channel 3 of drawer 248 has status 0x8003!
13:57:32 - HVPA channel 3 in module 248 is locked!
13:57:41 - HVPA channel 6 of drawer 91 has status 0x0003!
13:57:41 - HV is locked in 1 channel(s), was in rise default in 0 channel(s) and was still in progress in 4 channel(s)
13:57:41 - Unlocking 1 channel(s)...
13:57:43 - Checking for high voltage locked pixels...
13:57:47 - Module 248 has 1 pixel(s) locked (HVPA status of 0x77)!
13:57:57 - HVPA channel 2 of drawer 86 has status 0x0003!
13:58:00 - HVPA channel 3 of drawer 248 has status 0x8003!
13:58:00 - HVPA channel 3 in module 248 is locked!
13:58:10 - HV is locked in 1 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
13:58:10 - Unlocking 1 channel(s)...
13:58:11 - Checking for high voltage locked pixels...
13:58:16 - Module 248 has 1 pixel(s) locked (HVPA status of 0x77)!
13:58:25 - HVPA channel 2 of drawer 86 has status 0x0003!
13:58:25 - HVPA channel 2 of drawer 192 has status 0x0003!
13:58:29 - HVPA channel 3 of drawer 248 has status 0x8003!
13:58:29 - HVPA channel 3 in module 248 is locked!
13:58:41 - HV is locked in 1 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
13:58:41 - Unlocking 1 channel(s)...
13:58:42 - Checking for high voltage locked pixels...
13:58:47 - Module 248 has 1 pixel(s) locked (HVPA status of 0x77)!
13:58:57 - HVPA channel 2 of drawer 86 has status 0x0003!
13:59:00 - HVPA channel 3 of drawer 248 has status 0x8003!
13:59:00 - HVPA channel 3 in module 248 is locked!
13:59:11 - HV is locked in 1 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
13:59:11 - Unlocking 1 channel(s)...
13:59:13 - Checking for high voltage locked pixels...
13:59:18 - Module 248 has 1 pixel(s) locked (HVPA status of 0x77)!
13:59:27 - HVPA channel 2 of drawer 86 has status 0x0003!
13:59:30 - HVPA channel 3 of drawer 248 has status 0x8003!
13:59:30 - HVPA channel 3 in module 248 is locked!
13:59:40 - HV is locked in 1 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
13:59:40 - Unlocking 1 channel(s)...
13:59:41 - Checking for high voltage locked pixels...
13:59:46 - Module 248 has 1 pixel(s) locked (HVPA status of 0x77)!
13:59:56 - HVPA channel 2 of drawer 86 has status 0x0003!
13:59:59 - HVPA channel 3 of drawer 248 has status 0x8003!
13:59:59 - HVPA channel 3 in module 248 is locked!
14:00:02 - HVPA channel 4 of drawer 128 has status 0x0003!
14:00:05 - HVPA channel 5 of drawer 53 has status 0x0003!
14:00:09 - HV is locked in 1 channel(s), was in rise default in 0 channel(s) and was still in progress in 4 channel(s)
14:00:09 - Unlocking 1 channel(s)...
14:00:09 - ("NmcError: Failed to write 'NectarFrontEnd_248.Hvpa.Hvpa3.Disable': status Bad statuses are present, code 0xfffffffb, status NmcStatus.Mismatch", 4294967291)
14:00:11 - Checking for high voltage locked pixels...
14:00:16 - Module 248 has 1 pixel(s) locked (HVPA status of 0x77)!
14:00:25 - HVPA channel 2 of drawer 86 has status 0x0003!
14:00:29 - HVPA channel 3 of drawer 248 has status 0x8003!
14:00:29 - HVPA channel 3 in module 248 is locked!
14:00:41 - HVPA channel 6 of drawer 63 has status 0x0003!
14:00:41 - HV is locked in 1 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
14:00:41 - Unlocking 1 channel(s)...
14:00:43 - Checking for high voltage locked pixels...
14:00:57 - HVPA channel 2 of drawer 86 has status 0x0003!
14:01:00 - HVPA channel 3 of drawer 248 has status 0x0003!
14:01:03 - HVPA channel 4 of drawer 208 has status 0x0003!
14:01:12 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
14:01:12 - Checking for high voltage locked pixels...
14:01:26 - HVPA channel 2 of drawer 86 has status 0x0003!
14:01:29 - HVPA channel 3 of drawer 248 has status 0x0003!
14:01:36 - HVPA channel 5 of drawer 56 has status 0x0003!
14:01:39 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
14:01:41 - Checking for high voltage locked pixels...
14:01:55 - HVPA channel 2 of drawer 86 has status 0x0003!
14:01:59 - HVPA channel 3 of drawer 248 has status 0x0003!
14:02:09 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:02:11 - Checking for high voltage locked pixels...
14:02:25 - HVPA channel 2 of drawer 86 has status 0x0003!
14:02:28 - HVPA channel 3 of drawer 248 has status 0x0003!
14:02:41 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:02:41 - Checking for high voltage locked pixels...
14:02:55 - HVPA channel 2 of drawer 86 has status 0x0003!
14:02:58 - HVPA channel 3 of drawer 248 has status 0x0003!
14:03:08 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:03:11 - Checking for high voltage locked pixels...
14:03:25 - HVPA channel 2 of drawer 86 has status 0x0003!
14:03:28 - HVPA channel 3 of drawer 248 has status 0x0003!
14:03:38 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:03:41 - Checking for high voltage locked pixels...
14:03:55 - HVPA channel 2 of drawer 86 has status 0x0003!
14:03:58 - HVPA channel 3 of drawer 248 has status 0x0003!
14:04:08 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:04:11 - Checking for high voltage locked pixels...
14:04:25 - HVPA channel 2 of drawer 86 has status 0x0003!
14:04:28 - HVPA channel 3 of drawer 248 has status 0x0003!
14:04:41 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:04:41 - Checking for high voltage locked pixels...
14:04:56 - HVPA channel 2 of drawer 86 has status 0x0003!
14:04:59 - HVPA channel 3 of drawer 248 has status 0x0003!
14:05:12 - HVPA channel 6 of drawer 91 has status 0x0003!
14:05:12 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
14:05:12 - Updating voltage widget...
14:05:34 - Checking for high voltage locked pixels...
14:05:36 - Module 91 has 1 pixel(s) locked (HVPA status of 0x3f)!
14:05:48 - HVPA channel 2 of drawer 86 has status 0x0003!
14:05:51 - HVPA channel 3 of drawer 248 has status 0x0003!
14:05:54 - HVPA channel 4 of drawer 24 has status 0x0003!
14:05:58 - HVPA channel 5 of drawer 228 has status 0x0003!
14:06:01 - HVPA channel 6 of drawer 91 has status 0x8003!
14:06:01 - HVPA channel 6 in module 91 is locked!
14:06:01 - HV is locked in 1 channel(s), was in rise default in 0 channel(s) and was still in progress in 5 channel(s)
14:06:01 - Unlocking 1 channel(s)...
14:06:02 - Checking for high voltage locked pixels...
14:06:17 - HVPA channel 2 of drawer 86 has status 0x0003!
14:06:20 - HVPA channel 3 of drawer 248 has status 0x0003!
14:06:29 - HVPA channel 6 of drawer 91 has status 0x0003!
14:06:29 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
14:06:30 - Checking status of calibration box...
14:06:31 - Calibration box is configured with 13 leds at a frequency of 300.0 Hz and a voltage of 10.00 V
14:06:31 - Checking for high voltage locked pixels...
14:06:45 - HVPA channel 2 of drawer 86 has status 0x0003!
14:06:48 - HVPA channel 3 of drawer 248 has status 0x0003!
14:06:55 - HVPA channel 5 of drawer 40 has status 0x0003!
14:06:58 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
14:06:58 - Updating FEB temperature widget...
14:07:03 - Checking for high voltage locked pixels...
14:07:18 - HVPA channel 2 of drawer 86 has status 0x0003!
14:07:21 - HVPA channel 3 of drawer 67 has status 0x0003!
14:07:22 - HVPA channel 3 of drawer 158 has status 0x0003!
14:07:22 - HVPA channel 3 of drawer 248 has status 0x0003!
14:07:31 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 4 channel(s)
14:07:31 - Checking for high voltage locked pixels...
14:07:45 - HVPA channel 2 of drawer 86 has status 0x0003!
14:07:48 - HVPA channel 3 of drawer 248 has status 0x0003!
14:07:58 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:07:58 - Starting NMC
14:07:58 - Clearing PPS error counters
14:08:11 - Resetting FEB counters...
14:08:15 - Starting all configured modules...
14:08:20 - Checking for high voltage locked pixels...
14:08:38 - HVPA channel 2 of drawer 86 has status 0x0003!
14:08:41 - HVPA channel 3 of drawer 248 has status 0x0003!
14:08:50 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:08:50 - Checking for high voltage locked pixels...
14:09:04 - HVPA channel 2 of drawer 86 has status 0x0003!
14:09:08 - HVPA channel 3 of drawer 248 has status 0x0003!
14:09:18 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:09:20 - Checking for high voltage locked pixels...
14:09:31 - HVPA channel 1 of drawer 102 has status 0x0003!
14:09:34 - HVPA channel 2 of drawer 86 has status 0x0003!
14:09:37 - HVPA channel 3 of drawer 248 has status 0x0003!
14:09:47 - HVPA channel 6 of drawer 15 has status 0x0003!
14:09:47 - HVPA channel 6 of drawer 86 has status 0x0003!
14:09:47 - HVPA channel 6 of drawer 150 has status 0x0003!
14:09:47 - HVPA channel 6 of drawer 161 has status 0x0003!
14:09:47 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 7 channel(s)
14:09:50 - Checking for high voltage locked pixels...
14:10:04 - HVPA channel 2 of drawer 86 has status 0x0003!
14:10:07 - HVPA channel 3 of drawer 248 has status 0x0003!
14:10:11 - HVPA channel 4 of drawer 31 has status 0x0003!
14:10:17 - HVPA channel 6 of drawer 91 has status 0x0003!
14:10:17 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 4 channel(s)
14:10:20 - Checking for high voltage locked pixels...
14:10:38 - HVPA channel 2 of drawer 86 has status 0x0003!
14:10:41 - HVPA channel 3 of drawer 248 has status 0x0003!
14:10:50 - HVPA channel 6 of drawer 91 has status 0x0003!
14:10:50 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
14:10:51 - Checking for high voltage locked pixels...
14:11:07 - HVPA channel 2 of drawer 86 has status 0x0003!
14:11:10 - HVPA channel 3 of drawer 248 has status 0x0003!
14:11:20 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:11:20 - Checking for high voltage locked pixels...
14:11:34 - HVPA channel 2 of drawer 86 has status 0x0003!
14:11:38 - HVPA channel 3 of drawer 248 has status 0x0003!
14:11:41 - HVPA channel 4 of drawer 19 has status 0x0003!
14:11:41 - HVPA channel 4 of drawer 99 has status 0x0003!
14:11:41 - HVPA channel 4 of drawer 101 has status 0x0003!
14:11:41 - HVPA channel 4 of drawer 113 has status 0x0003!
14:11:41 - HVPA channel 4 of drawer 136 has status 0x0003!
14:11:41 - HVPA channel 4 of drawer 137 has status 0x0003!
14:11:41 - HVPA channel 4 of drawer 156 has status 0x0003!
14:11:41 - HVPA channel 4 of drawer 219 has status 0x0003!
14:11:47 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 10 channel(s)
14:11:50 - Checking for high voltage locked pixels...
14:12:04 - HVPA channel 2 of drawer 86 has status 0x0003!
14:12:07 - HVPA channel 3 of drawer 248 has status 0x0003!
14:12:17 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 2 channel(s)
14:12:20 - Checking for high voltage locked pixels...
14:12:22 - Module 91 has 1 pixel(s) locked (HVPA status of 0x3f)!
14:12:35 - HVPA channel 2 of drawer 86 has status 0x0003!
14:12:38 - HVPA channel 3 of drawer 248 has status 0x0003!
14:12:44 - HVPA channel 5 of drawer 228 has status 0x0003!
14:12:47 - HVPA channel 6 of drawer 91 has status 0x8003!
14:12:47 - HVPA channel 6 in module 91 is locked!
14:12:47 - HV is locked in 1 channel(s), was in rise default in 0 channel(s) and was still in progress in 4 channel(s)
14:12:47 - Unlocking 1 channel(s)...
14:12:50 - Checking for high voltage locked pixels...
14:13:04 - HVPA channel 2 of drawer 86 has status 0x0003!
14:13:07 - HVPA channel 3 of drawer 248 has status 0x0003!
14:13:18 - HVPA channel 6 of drawer 91 has status 0x0003!
14:13:18 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
14:13:18 - Stopped periodic check of locked high voltages
14:13:25 - Checking UCTS clock...
14:13:26 - UCTS and control PC clocks differ by 4.2 seconds
14:13:26 - Configuring UCTS with (10.200.7.250, 10.200.4.100, 44:a8:42:44:32:c9)
14:13:26 - Configuring TIB...
14:13:26 - Setting TIB forced mode to False
14:13:26 - Writing False to Camera.TIB.Forced.ForcedMode.ForcedMode_v
14:13:26 - Status: TibState.PpsStopped
14:13:26 - Alarms: AlarmVector.BufferFull|AlmostFull
14:13:26 - Connection status: False TXclient_alive: False
14:13:26 - Entering set_external_delay
14:13:26 - Setting ExternalDelayLocal to 27.000 ns
14:13:26 - Writing 27.0 to Camera.TIB.ExternalDelay.ExternalDelayLocal.ExternalDelayLocal_v
14:13:27 - Entering set_external_delay
14:13:27 - Setting ExternalDelayCalibration to 58.000 ns
14:13:27 - Writing 58.0 to Camera.TIB.ExternalDelay.ExternalDelayCalibration.ExternalDelayCalibration_v
14:13:27 - Entering set_external_delay
14:13:27 - Setting ExternalDelaySphe to 150.000 ns
14:13:27 - Writing 150.0 to Camera.TIB.ExternalDelay.ExternalDelaySphe.ExternalDelaySphe_v
14:13:27 - Setting TIB trigger mode to 0x54 = TriggerMask.SlowControl|Software|Calibration
14:13:27 - Writing 84 to Camera.TIB.TriggerMode.ModeMask.ModeMask_v
14:13:27 - Writing 1 to Camera.TIB.InputsEnable.InputMask.InputMask_v
14:13:27 - Writing False to Camera.TIB.BUSY.DisableBusy.DisableBusy_v
14:13:27 - Writing 0.0 to Camera.TIB.BUSY.ExtraDelayBusy.ExtraDelayBusy_v
14:13:27 - Writing True to Camera.TIB.Throttler.DisableThrottler.DisableThrottler_v
14:13:27 - Writing 255 to Camera.TIB.Throttler.MaxTriggers50us.MaxTriggers50us_v
14:13:27 - Writing 255 to Camera.TIB.Throttler.MaxTriggers1ms.MaxTriggers1ms_v
14:13:27 - Status: TibState.PpsStopped
14:13:27 - Alarms: AlarmVector.BufferFull|AlmostFull
14:13:27 - Connection status: False TXclient_alive: False
14:13:27 - Writing False to Camera.TIB.Pedestal.PeriodicPedestalEnable.PeriodicPedestalEnable_v
14:13:27 - Writing 100 to Camera.TIB.Pedestal.PeriodicPedestalFrequency.PeriodicPedestalFrequency_v
14:13:27 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:13:27 - Writing False to Camera.TIB.Pedestal.DigitalPedestalEnable.DigitalPedestalEnable_v
14:13:27 - Writing 100 to Camera.TIB.Pedestal.DigitalPedestal_config_Rate.DigitalPedestal_config_Rate_v
14:13:27 - Writing 375764 to Camera.TIB.Pedestal.DigitalPedestalSeed.DigitalPedestalSeed_v
14:13:27 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:13:27 - Writing False to Camera.TIB.Pedestal.AnalogPedestalEnable.AnalogPedestalEnable_v
14:13:27 - Writing 3000.0 to Camera.TIB.Pedestal.AnalogPedestalThreshold.AnalogPedestalThreshold_v
14:13:27 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:13:31 - Connecting TIB to camera server at 10.10.4.12:222
14:13:31 - Starting TIB...
14:13:31 - Status: TibState.PpsStopped
14:13:31 - Alarms: AlarmVector.BufferFull|AlmostFull
14:13:31 - Connection status: False TXclient_alive: False
14:13:31 - Resetting run
14:13:31 - Wait for UCTS external trigger before enabling trigger...
14:13:32 - Status: TibState.Standby
14:13:32 - Alarms: AlarmVector.NoAlarm
14:13:32 - Connection status: False TXclient_alive: True
14:13:32 - Status: TibState.Standby
14:13:32 - Alarms: AlarmVector.NoAlarm
14:13:32 - Connection status: False TXclient_alive: True
14:13:33 - Status: TibState.Armed
14:13:33 - Alarms: AlarmVector.NoAlarm
14:13:33 - Connection status: False TXclient_alive: True
14:14:12 - Starting TIB triggers...
14:14:12 - Enabling trigger
14:14:12 - Status: TibState.Running
14:14:12 - Alarms: AlarmVector.NoAlarm
14:14:12 - Connection status: False TXclient_alive: True
14:14:14 - Status: TibState.Running ConnectionStatus: False TXclient_alive: True
14:14:14 - Alarms: AlarmVector.NoAlarm
14:14:14 - Temperature: 41.06
14:14:14 - Local camera rate: 0.0 Hz
14:14:14 - Stereo trigger rate: 0.0 Hz
14:14:14 - SPE trigger rate: 0.0 Hz
14:14:14 - Calibration trigger rate: 0.0 Hz
14:14:14 - Software trigger rate: 0.0 Hz
14:14:14 - Pedestal trigger rate: 0.0 Hz
14:14:14 - Collected trigger rate: 0.0 Hz
14:14:14 - Busy rate: 0.0 Hz
14:14:14 - Camera trigger rate: 0.0 Hz
14:14:24 - Configuring calibration box...
14:14:24 - Configuring calibration box with leds=0x1fff voltage=10.00 frequency=1000.0/1 central_current=0.0 duration=0 width=1 lemo_out=True fiber1_out=True fiber2_out=False lemo_in=False enabled=False
14:14:25 - Starting flashes...
14:14:27 - Stopping flashes...
14:14:45 - Status: TibState.Running ConnectionStatus: False TXclient_alive: True
14:14:45 - Alarms: AlarmVector.BufferFull|AlmostFull
14:14:45 - Temperature: 41.06
14:14:45 - Local camera rate: 0.0 Hz
14:14:45 - Stereo trigger rate: 0.0 Hz
14:14:45 - SPE trigger rate: 0.0 Hz
14:14:45 - Calibration trigger rate: 0.0 Hz
14:14:45 - Software trigger rate: 0.0 Hz
14:14:45 - Pedestal trigger rate: 0.0 Hz
14:14:45 - Collected trigger rate: 0.0 Hz
14:14:45 - Busy rate: 0.0 Hz
14:14:45 - Camera trigger rate: 0.0 Hz
14:14:55 - Configuring calibration box...
14:14:55 - Configuring calibration box with leds=0x1fff voltage=10.00 frequency=1000.0/1 central_current=0.0 duration=0 width=1 lemo_out=True fiber1_out=True fiber2_out=False lemo_in=False enabled=False
14:14:56 - Starting flashes...
14:14:59 - Stopping TIB...
14:14:59 - Status: TibState.Running
14:14:59 - Alarms: AlarmVector.BufferFull|AlmostFull
14:14:59 - Connection status: False TXclient_alive: True
14:14:59 - Disabling trigger
14:14:59 - Disabling PPS
14:15:01 - Status: TibState.PpsStopped
14:15:01 - Alarms: AlarmVector.BufferFull|AlmostFull
14:15:01 - Connection status: False TXclient_alive: True
14:15:02 - Closing TIB connection to camera server
14:15:14 - Checking UCTS clock...
14:15:14 - UCTS and control PC clocks differ by 3.3 seconds
14:15:14 - Configuring UCTS with (10.200.7.250, 10.200.4.100, 44:a8:42:44:32:c9)
14:15:14 - Configuring TIB...
14:15:14 - Setting TIB forced mode to False
14:15:14 - Writing False to Camera.TIB.Forced.ForcedMode.ForcedMode_v
14:15:14 - Status: TibState.PpsStopped
14:15:14 - Alarms: AlarmVector.BufferFull|AlmostFull
14:15:14 - Connection status: False TXclient_alive: False
14:15:14 - Entering set_external_delay
14:15:14 - Setting ExternalDelayLocal to 27.000 ns
14:15:14 - Writing 27.0 to Camera.TIB.ExternalDelay.ExternalDelayLocal.ExternalDelayLocal_v
14:15:14 - Entering set_external_delay
14:15:14 - Setting ExternalDelayCalibration to 58.000 ns
14:15:14 - Writing 58.0 to Camera.TIB.ExternalDelay.ExternalDelayCalibration.ExternalDelayCalibration_v
14:15:14 - Entering set_external_delay
14:15:14 - Setting ExternalDelaySphe to 150.000 ns
14:15:14 - Writing 150.0 to Camera.TIB.ExternalDelay.ExternalDelaySphe.ExternalDelaySphe_v
14:15:14 - Setting TIB trigger mode to 0x54 = TriggerMask.SlowControl|Software|Calibration
14:15:14 - Writing 84 to Camera.TIB.TriggerMode.ModeMask.ModeMask_v
14:15:14 - Writing 1 to Camera.TIB.InputsEnable.InputMask.InputMask_v
14:15:14 - Writing False to Camera.TIB.BUSY.DisableBusy.DisableBusy_v
14:15:15 - Writing 0.0 to Camera.TIB.BUSY.ExtraDelayBusy.ExtraDelayBusy_v
14:15:15 - Writing True to Camera.TIB.Throttler.DisableThrottler.DisableThrottler_v
14:15:15 - Writing 255 to Camera.TIB.Throttler.MaxTriggers50us.MaxTriggers50us_v
14:15:15 - Writing 255 to Camera.TIB.Throttler.MaxTriggers1ms.MaxTriggers1ms_v
14:15:15 - Status: TibState.PpsStopped
14:15:15 - Alarms: AlarmVector.BufferFull|AlmostFull
14:15:15 - Connection status: False TXclient_alive: False
14:15:15 - Writing False to Camera.TIB.Pedestal.PeriodicPedestalEnable.PeriodicPedestalEnable_v
14:15:15 - Writing 100 to Camera.TIB.Pedestal.PeriodicPedestalFrequency.PeriodicPedestalFrequency_v
14:15:15 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:15:15 - Writing False to Camera.TIB.Pedestal.DigitalPedestalEnable.DigitalPedestalEnable_v
14:15:15 - Writing 100 to Camera.TIB.Pedestal.DigitalPedestal_config_Rate.DigitalPedestal_config_Rate_v
14:15:15 - Writing 375764 to Camera.TIB.Pedestal.DigitalPedestalSeed.DigitalPedestalSeed_v
14:15:15 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:15:15 - Writing False to Camera.TIB.Pedestal.AnalogPedestalEnable.AnalogPedestalEnable_v
14:15:15 - Writing 3000.0 to Camera.TIB.Pedestal.AnalogPedestalThreshold.AnalogPedestalThreshold_v
14:15:15 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:15:15 - Connecting TIB to camera server at 10.10.4.12:222
14:15:16 - Starting TIB...
14:15:16 - Status: TibState.PpsStopped
14:15:16 - Alarms: AlarmVector.BufferFull|AlmostFull
14:15:16 - Connection status: False TXclient_alive: True
14:15:16 - Resetting run
14:15:16 - Wait for UCTS external trigger before enabling trigger...
14:15:16 - Status: TibState.Standby
14:15:16 - Alarms: AlarmVector.NoAlarm
14:15:16 - Connection status: False TXclient_alive: True
14:15:16 - Status: TibState.Standby
14:15:16 - Alarms: AlarmVector.NoAlarm
14:15:16 - Connection status: False TXclient_alive: True
14:15:17 - Status: TibState.Armed
14:15:17 - Alarms: AlarmVector.NoAlarm
14:15:17 - Connection status: False TXclient_alive: True
14:15:18 - Starting TIB triggers...
14:15:18 - Enabling trigger
14:15:18 - Status: TibState.Running
14:15:18 - Alarms: AlarmVector.NoAlarm
14:15:18 - Connection status: False TXclient_alive: True
14:15:19 - Status: TibState.Running ConnectionStatus: False TXclient_alive: True
14:15:19 - Alarms: AlarmVector.NoAlarm
14:15:19 - Temperature: 41.12
14:15:19 - Local camera rate: 0.0 Hz
14:15:19 - Stereo trigger rate: 0.0 Hz
14:15:19 - SPE trigger rate: 3.0 Hz
14:15:19 - Calibration trigger rate: 999.0 Hz
14:15:19 - Software trigger rate: 0.0 Hz
14:15:19 - Pedestal trigger rate: 0.0 Hz
14:15:19 - Collected trigger rate: 999.0 Hz
14:15:19 - Busy rate: 0.0 Hz
14:15:19 - Camera trigger rate: 999.0 Hz
14:15:21 - Stopping TIB...
14:15:22 - Status: TibState.Running
14:15:22 - Alarms: AlarmVector.BufferFull|AlmostFull
14:15:22 - Connection status: False TXclient_alive: True
14:15:22 - Disabling trigger
14:15:22 - Disabling PPS
14:15:23 - Status: TibState.PpsStopped
14:15:23 - Alarms: AlarmVector.BufferFull|AlmostFull
14:15:23 - Connection status: False TXclient_alive: True
14:15:24 - Closing TIB connection to camera server
14:15:26 - Stopping NMC
14:15:31 - Found 0 PPS errors.
14:15:32 - Starting task "Check high voltages"
14:15:37 - Found 265 drawer(s) in Nominal state, 0 in Low state and 0 in Wait state
14:16:25 - Actual voltage is between 0.00 V and 1124.23 V.
14:16:30 - Low voltages range between 100.00 V and 100.00 V. 
14:17:38 - Reference voltages range between 840.02 V and 1124.57 V. 
14:18:31 - Maximum difference between expected and actual voltages is 995.27 V
14:19:05 - Maximum of current limits is 6290.01 uA
14:19:05 - Checking status of HVPA module in 265 drawer(s)...
14:19:23 - HVPA channel 2 of drawer 86 has status 0x0003!
14:19:26 - HVPA channel 3 of drawer 248 has status 0x0003!
14:19:35 - HVPA channel 6 of drawer 91 has status 0x0003!
14:19:35 - HV is locked in 0 channel(s), was in rise default in 0 channel(s) and was still in progress in 3 channel(s)
14:19:35 - Checking reference and low voltages against /data/config/2021/20210830/config_cameraqm_265febs_newDACL.NMC.xml
14:20:14 - Done
14:20:14 - Finished task "Check high voltages"
14:22:50 - Stopping NMC
14:22:56 - Found 0 PPS errors.
14:22:58 - Status: TibState.PpsStopped
14:22:58 - Alarms: AlarmVector.BufferFull|AlmostFull
14:22:58 - Connection status: False TXclient_alive: False
14:22:59 - External delays: Local=27.000 ns, Calibration=58.000 ns, Sphe=150.000 ns
14:22:59 - Trigger mask: 0x54 = TriggerMask.SlowControl|Software|Calibration
14:22:59 - Input mask: 0x1
14:22:59 - Disable BUSY: False
14:22:59 - Extra BUSY Delay: 0.0
14:22:59 - Temperature: 41.44
14:23:16 - Starting NMC
14:23:16 - Clearing PPS error counters
14:23:29 - Resetting FEB counters...
14:23:33 - Starting all configured modules...
14:23:52 - Calling method FSM.gotoObserving in object FSM
14:23:52 - Starting run 2564
14:24:27 - Configuring calibration box...
14:24:27 - Configuring calibration box with leds=0x1fff voltage=10.00 frequency=1000.0/1 central_current=0.0 duration=0 width=1 lemo_out=True fiber1_out=True fiber2_out=False lemo_in=False enabled=False
14:24:29 - Starting flashes...
14:24:34 - Checking UCTS clock...
14:24:34 - UCTS and control PC clocks differ by 3.5 seconds
14:24:34 - Configuring UCTS with (10.200.7.250, 10.200.4.100, 44:a8:42:44:32:c9)
14:24:34 - Configuring TIB...
14:24:34 - Setting TIB forced mode to False
14:24:34 - Writing False to Camera.TIB.Forced.ForcedMode.ForcedMode_v
14:24:34 - Status: TibState.PpsStopped
14:24:34 - Alarms: AlarmVector.BufferFull|AlmostFull
14:24:34 - Connection status: False TXclient_alive: False
14:24:34 - Entering set_external_delay
14:24:34 - Setting ExternalDelayLocal to 27.000 ns
14:24:34 - Writing 27.0 to Camera.TIB.ExternalDelay.ExternalDelayLocal.ExternalDelayLocal_v
14:24:34 - Entering set_external_delay
14:24:34 - Setting ExternalDelayCalibration to 58.000 ns
14:24:34 - Writing 58.0 to Camera.TIB.ExternalDelay.ExternalDelayCalibration.ExternalDelayCalibration_v
14:24:34 - Entering set_external_delay
14:24:34 - Setting ExternalDelaySphe to 150.000 ns
14:24:34 - Writing 150.0 to Camera.TIB.ExternalDelay.ExternalDelaySphe.ExternalDelaySphe_v
14:24:35 - Setting TIB trigger mode to 0x54 = TriggerMask.SlowControl|Software|Calibration
14:24:35 - Writing 84 to Camera.TIB.TriggerMode.ModeMask.ModeMask_v
14:24:35 - Writing 1 to Camera.TIB.InputsEnable.InputMask.InputMask_v
14:24:35 - Writing False to Camera.TIB.BUSY.DisableBusy.DisableBusy_v
14:24:35 - Writing 0.0 to Camera.TIB.BUSY.ExtraDelayBusy.ExtraDelayBusy_v
14:24:35 - Writing True to Camera.TIB.Throttler.DisableThrottler.DisableThrottler_v
14:24:35 - Writing 255 to Camera.TIB.Throttler.MaxTriggers50us.MaxTriggers50us_v
14:24:35 - Writing 255 to Camera.TIB.Throttler.MaxTriggers1ms.MaxTriggers1ms_v
14:24:35 - Status: TibState.PpsStopped
14:24:35 - Alarms: AlarmVector.BufferFull|AlmostFull
14:24:35 - Connection status: False TXclient_alive: False
14:24:35 - Writing False to Camera.TIB.Pedestal.PeriodicPedestalEnable.PeriodicPedestalEnable_v
14:24:35 - Writing 100 to Camera.TIB.Pedestal.PeriodicPedestalFrequency.PeriodicPedestalFrequency_v
14:24:35 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:24:35 - Writing False to Camera.TIB.Pedestal.DigitalPedestalEnable.DigitalPedestalEnable_v
14:24:35 - Writing 100 to Camera.TIB.Pedestal.DigitalPedestal_config_Rate.DigitalPedestal_config_Rate_v
14:24:35 - Writing 375764 to Camera.TIB.Pedestal.DigitalPedestalSeed.DigitalPedestalSeed_v
14:24:35 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:24:35 - Writing False to Camera.TIB.Pedestal.AnalogPedestalEnable.AnalogPedestalEnable_v
14:24:35 - Writing 3000.0 to Camera.TIB.Pedestal.AnalogPedestalThreshold.AnalogPedestalThreshold_v
14:24:35 - Writing False to Camera.TIB.Pedestal.Fake_trig_enable.Fake_trig_enable_v
14:24:36 - Connecting TIB to camera server at 10.10.4.12:222
14:24:36 - Starting TIB...
14:24:36 - Status: TibState.PpsStopped
14:24:36 - Alarms: AlarmVector.BufferFull|AlmostFull
14:24:36 - Connection status: False TXclient_alive: True
14:24:36 - Resetting run
14:24:36 - Wait for UCTS external trigger before enabling trigger...
14:24:36 - Status: TibState.Standby
14:24:36 - Alarms: AlarmVector.NoAlarm
14:24:36 - Connection status: False TXclient_alive: True
14:24:36 - Status: TibState.Standby
14:24:36 - Alarms: AlarmVector.NoAlarm
14:24:36 - Connection status: False TXclient_alive: True
14:24:37 - Status: TibState.Armed
14:24:37 - Alarms: AlarmVector.NoAlarm
14:24:38 - Connection status: True TXclient_alive: True
14:24:39 - Starting TIB triggers...
14:24:39 - Enabling trigger
14:24:39 - Status: TibState.Running
14:24:39 - Alarms: AlarmVector.ReadError
14:24:39 - Connection status: True TXclient_alive: True
14:24:42 - Status: TibState.Running ConnectionStatus: True TXclient_alive: True
14:24:42 - Alarms: AlarmVector.ReadError
14:24:42 - Temperature: 41.50
14:24:42 - Local camera rate: 0.0 Hz
14:24:42 - Stereo trigger rate: 0.0 Hz
14:24:42 - SPE trigger rate: 0.0 Hz
14:24:42 - Calibration trigger rate: 999.0 Hz
14:24:42 - Software trigger rate: 0.0 Hz
14:24:42 - Pedestal trigger rate: 0.0 Hz
14:24:42 - Collected trigger rate: 999.0 Hz
14:24:42 - Busy rate: 0.0 Hz
14:24:42 - Camera trigger rate: 999.0 Hz
14:25:31 - Stopping TIB...
14:25:31 - Status: TibState.Running
14:25:31 - Alarms: AlarmVector.ReadError
14:25:31 - Connection status: True TXclient_alive: True
14:25:31 - Disabling trigger
14:25:31 - Disabling PPS
14:25:32 - Status: TibState.PpsStopped
14:25:32 - Alarms: AlarmVector.ReadError
14:25:32 - Connection status: True TXclient_alive: True
14:25:33 - Closing TIB connection to camera server
14:25:34 - Status: TibState.PpsStopped
14:25:34 - Alarms: AlarmVector.ReadError
14:25:34 - Connection status: False TXclient_alive: False
14:25:34 - Run duration: 0:00:54
14:25:34 - Local camera triggers: 0
14:25:34 - Slow control triggers: 0
14:25:34 - Single photo-electron triggers: 0
14:25:34 - Calibration triggers: 53247
14:25:34 - Software triggers from UCTS: 0
14:25:34 - Pedestal triggers: 0
14:25:34 - Collected input triggers: 51913
14:25:34 - Busy triggers: 0
14:25:34 - Camera event triggers sent to clusters: 51913
14:25:39 - Stopping flashes...
14:26:14 - Calling method FSM.gotoReady in object FSM
14:26:21 - Stopping NMC
14:26:26 - Found 0 PPS errors.
14:26:28 - NMC is in state NmcState.Configured and knows of 265 drawer(s)
14:26:42 - All front-end boards have counted 51913 event(s)
14:26:48 - Found 0 PPS errors.
14:26:51 - Event builder is in state CameraServerState.Ready and has output 51913 event(s) during run #2565.