Tags:
create new tag
view all tags

TRT Whiteboard

Last reloaded at 22:17:23 on 20 Jan 2021.
TRT Phone Numbers at the bottom of the page.

This Whiteboard must reflect the actual state of the detector parts. Obsolete entries have to be removed by the author or by the person who made the fixes/changes.
Never mask a TRT alarm without agreement from the TRT Run Coordinator or the TRT DCS Expert.

Jump to: M6 · General · HV · GGSS · LV · DAQ · Monitoring · Active Gas · Cooling and Temperature · DCS and Alarms · HWI · Other

Recent ATLAS Runs

Data Quality

Shifter Links

TRT Shifter Instructions
ATLAS e-log
ATLAS Run Status
LHC Page 1, Logbook
ID Operations Database
High-Voltage Trip Record

General Links

Program of the Day
ATLAS Detector Operation
ATLAS Detector Status (DCS)
Current ATLAS Shift Crew
TRT General Shifter Instructions (ppt)
TRT DCS Shifter Instructions (pdf)

M6 Comments

Date Author Location Comment
14 October 2014 14:27 Andrey Loginov Remote TRT operates with some of the temperatures slightly above warning levels. Monitor the trends / make sure they are flat (temperatures are not rising).
14 October 2014 14:27 Andrey Loginov Remote Ignore the "1 connection lost" error (pcatltrthva) on the main DCS FSM screen
14 October 2014 14:27 Andrey Loginov Remote Ignore the 2 warnings / 1 fatal coming from HWI (they are on the ATLAS Alarm Screen)

M5 Comments

Date Author Location Comment
11 September 2014 14:27 Andrey Loginov Remote Ignore the CO2 pump "NOT READY" error -- there is no need to call experts on this one
11 September 2014 14:27 Andrey Loginov Remote TRT operates with some of the temperatures slightly above warning levels. Monitor the trends / make sure they are flat (temperatures are not rising).
11 September 2014 14:27 Andrey Loginov ACR Operating conditions:
- CO2 pump is "NOT READY" (doesn't affect anything)
- Ar active gas mixture is used
- TRT DAQ fw development work is ongoing

M3 Comments

Date Author Location Comment
20 May 2014 16:27 Andrey Loginov ACR TRT operates with some of the temperatures slightly above warning levels. Monitor the trends / make sure they are flat (temperatures are not rising).
20 May 2014 16:34 Andrey Loginov ACR Operating conditions:
- GGSS is disabled
- Ar active gas mixture is used
- TRT DAQ development/tests are ongoing

General Comments

Date Author Location Comment
04 Oct 2009 09:55 James Degenhardt TRT ACR Desk Do not run DB Explorer at the TRT ACR Desk. It may freeze the machine.
02 Apr 2010 09:48 Dominick Olivito TRT Calibration Schedule Click on TRT Calibration at the very top of this page to see the calibration schedule. If a scan is scheduled during your shift, click on the scan name to go to the relevant instructions.

The ATLAS combined run (even without beam) always has priority over our calibrations! They should only be performed during dedicated calibration periods. If there is going to be a test run using the ATLAS partition during the calibration period, please ask the Shift Leader to have the TRT removed before starting calibrations.
03 May 2010 14:40 James Degenhardt TRT Shift Summary Template Please use the TRT Shift Summary Template to write your shift summaries.
16 May 2011 16:30 Jonathan Stahlman TRT DAQ Request Please note that an ERROR level MRS message almost always means that a manual action is needed (either by the expert or the shifter). If you see an ERROR message and don't know what to do, please call the DAQ expert in addition to reporting it in the e-log.

High Voltage System

If you encounter HV trips, please note them down on the High-Voltage Trip Record.
Only permanent changes should be noted in the following table.

Date Author Location Comment
27 Apr 2009 09:43 Anatoli Romaniouk HVA S19S20 WA4 1T Off forever. Short on the line.
03 May 2010 14:55 Anatoli Romaniouk HVB S19 M3 A2 Off forever. Short on the line.
30 May 2015 17:13 Konstantin Zhukov HVB S9 M3 A5 Crate1/Branch1/Cell 018 was checked. no shortcut was found. it works well.
30 May 2015 17:17 Konstantin Zhukov HVB S24.M3.B1 Crate2/Branch2/Cell 079. There was a shortcut. Fuse was blown succesfully. Restored. it works well.
30 May 2015 17:20 Konstantin Zhukov HVB S26 M3 B1 Crate2/Branch2/Cell 100. It was checked. no shortcut was found. it works well.
30 May 2015 17:22 Konstantin Zhukov HVB S3 M3 B4 Crate1/Branch0/Cell 040. It was checked. No shortcut was found. When cell is going from Prepared to Safe it tripped. If manually switch it from Prepared to 1000V it works. No errors. aftert it it goes to SAFE and to READY with no problem.
13.07.2015
it was decided to change PREPARED state to 950V. The cell was tripping when going from PREPARED state to SAFE state. Probably there is something in this HV line. need to be watching during wark.
13 Jul 2015 10:20 Konstantin Zhukov HVB S21 M1 B2 Crate1/Branch2/cell 027
The cell tripped. We can see on output voltage and output current trends there was some activity before it's tripping. While normal current is about 20mkA there are spikes in current up to 120mkA and changing of the voltage for ~40V down. it can point at starting of shortages when currents increases to threshold level and HV cell is trying to keep current limited by making voltage lower. to be wathcing.
13 Jul 2015 10:35 Konstantin Zhukov HVB S17 M3 B5 Crate1/Branch3/Cell 049
in June we found the real volatge is less the Uset for 200V. It appeared contact between cell and HV LEMO was poor. It was successfully fixed. Work OK.
13 Jul 2015 10:39 Konstantin Zhukov HVB S4 M2 A1 Crate2/Branch0/Cell 029 HVM 2.0.2
in June we found the real voltage is less the Uset for 400V. Problem was in contact between cell and HV LEMO connector. Fixed. HVM module was changed to HVM 8.0.2.
13 Jul 2015 10:53 Konstantin Zhukov HVB S12 M1 B1 Crate2/Branch1/Cell 026
Real voltage was below setpoint for 50V. The cell was recalibrated.
13 Jul 2015 10:57 Konstantin Zhukov HVB S18 M1 B2 Crate2/Branch2/Cell 006 HVM 2.2.1
There was no contact between HV cell and LEMO HV connector.
HVM module was changed to HVM 8.1.1.
Contact was fixed successfully. cell is ok.
04 Aug 2015 15:30 Konstantin Zhukov HVA S21S22 WB1 B Crate4/Branch1/Cell089:It was not possible to switch on the cell. At 25V current was 250mkA => shortcut of the wire to the straw. Fuse was blown. Line was recovered.
13 Apr 2016 17:19 Konstantin Zhukov HVB S1 M2 A1 Crate1/Branch0/Cell008
double error status. Cell was changed. new calibration is implemented.
13 Apr 2016 17:23 Konstantin Zhukov HVB S9 M3 B3 Crate1/Branch1/Cell021
double error status. Cell was changed. new calibration is implemented.
13 Apr 2016 17:25 Konstantin Zhukov HVB S19 M3 A2 Crate1/Branch2/Cell015
Cell was OFF.
double error status. Cell was changed. new calibration is implemented.
13 Apr 2016 17:26 Konstantin Zhukov HVB S31 M2 A2 Crate1/Branch3/Cell030
double error status. Cell was changed. new calibration is implemented.
13 Apr 2016 17:28 Konstantin Zhukov HVC S19S20 WB2 B Crate6/Branch0/Cell103
cell was OFF.
double error status. Cell was changed. new calibration is implemented.
13 Apr 2016 17:30 Konstantin Zhukov HVC S9S10 WB6 B Crate6/Branch0/Cell016
double error status. Cell was changed. new calibration is implemented.
13 Apr 2016 17:33 Konstantin Zhukov HVA S19S20 WA4 1T Crate3/Branch2/Cell039
Cell was OFF (couldn't be switch ON)
double error status. Cell was changed. new calibration is implemented.
13 Apr 2016 17:34 Konstantin Zhukov HVA S21S22 WB2 B Crate4/Branch1/Cell092
Cell was OFF (couldn't be switch ON)
double error status. Cell was changed. new calibration is implemented.
01 Jul 2016 11:10 Konstantin Zhukov HVB S30 M1 A2 Crate2/Branch3/Cell024
Cell tripped. when trying to switch off it's tripping.
Shortcut found when tested with other HV supply.
Fuse was burned successfully. HV channel works well.
17 Aug 2016 08:55 Konstantin Zhukov HVB S15 M3 A2 Crate1/Branch1/Cell078
Cell tripping: it couldn't be switch to PREPARED.
When line tested with other HV supply CAEN: trip at 200V, >2mA.
Fuse was burned successfully with one try. HV channel switched to READY. OK.
17 Aug 2016 09:50 Konstantin Zhukov HVB S28 M3 A4 Crate2/Branch3/Cell017
the cell is tripping. Also S28M3A4 is connected to line S28M3A2 (crate2/branch3/cell015).
Both of measured lines are tripping at low voltage ~100V.
line S28M3A4: fuse was burned and both of lines can work.
S28M3A4 and S28M3A2 set in READY. OK.
06 Sep 2016 11:28 Konstantin Zhukov HVB S15 M3 A3 Crate1/Branch1/Cell076
The line is tripping at 160V.
Fuse was blown with one try. Line is recovered successfully. it's in READY.
28 Sep 2016 11:33 Konstantin Zhukov HVB S9 M3 B5 (really S15 M3 B5) Crate1/Branch3/Cell048 was connected to Crate1/Branch3/Cell051 and vise verse.

REALLY it was line S15 M3 B5 Crate1/Branch3/Cell051.
the line was tripped. After recovering - tripped again.
Line was tested: it's tripping at ~300V.
Fuse was blown successfully with one try. Line is recovered. It's in READY.
27 Aug 2017 20:30 Konstantin Zhukov HVB S9 M3 A1 Crate1/Branch1/Cell014
Line is tripping at 1400V.
Fuse was blown with one try. Line is recovered successfully. Cell is READY.
17 Apr 2018 18:10 Konstantin Zhukov HVB S15 M3 A5 Crate1/Branch1/Cell081
Line is tripping.
Fuse was blown by Anatoli. Line is recovered successfully. Cell is READY.
17 Apr 2018 18:12 Konstantin Zhukov HVB S22 M5 B5 Crate2/Branch3/Cell075
Line is tripping.
Fuse was blown by Anatoli. Line is recovered successfully. Cell is READY.
17 Apr 2018 18:12 Konstantin Zhukov HVA S11S12 WA6 2B Crate3/Branch3/Cell032
Line is tripping.
Fuse was blown by Anatoli. Line is recovered successfully. Cell is READY.
17 Apr 2018 18:15 Konstantin Zhukov HVB S22 M3 A4 Crate2/Branch2/Cell059
Line is tripping at 600V.
Fuse was blown by Kostya. Line is recovered successfully. Cell is READY.
17 May 2018 07:25 Konstantin Zhukov HVB S6 M1 A1 Crate2/Branch0/Cell044
Line is tripping at 1100V.
Fuse was blown by Kostya with one try at 1450V. Line is recovered successfully.
Cell is READY.
11 Jun 2018 13:00 Konstantin Zhukov HVB S28 M3 A5 Crate2/Branch3/Cell018
line was tripped on 09/06/2018 at ~5am.
Anatoli was trying to restore it - not possible.
Fuse was blown by Anatoli. Line is working but voltage is fluctuating +-4V.
Cell is READY
31 Aug 2018 15:28 Konstantin Zhukov HVB S13 M1 B1 Crate1/Branch1/Cell047
Line is tripping at ~700V.
Fuse was blown by Kostya with try at 1500V (1450V didn't work). Line is recovered successfully.
Cell is READY.

Gas Gain Stabilization System

Date Author Location Comment
18 Aug 2009 14:35 Anatoli Romaniouk GGSS GGSS activated for the whole TRT.

Low Voltage System

Date Author Location Comment
27 Apr 2009 09:59 Jim Degenhardt Endcap A Slice 1 WA2 (DAQ sector 32) Two boards permanently dead due to analog short (FSM state MIXED)
27 Apr 2009 09:59 Jim Degenhardt Endcap A Slice 18 WB5 (DAQ sector 17) One board permanently dead due to analog short (FSM state MIXED)
27 Apr 2009 09:59 Jim Degenhardt Endcap A Slice 25 WA2 (DAQ sector 24) Two boards permanently dead due to analog short (FSM state MIXED)
27 Apr 2009 09:59 Jim Degenhardt Endcap C Slice 26 WA2 (DAQ sector 25) Two boards permanently dead due to analog short (FSM state MIXED)

DAQ

Date Author Location Comment
29 Jun 2009 10:07 Dominick Olivito Barrel C Stack 3 Board 3S2 (triangle in module 3) is permanently dead due to a LV issue. This has 0% occupancy.
29 Jun 2009 10:08 Dominick Olivito Endcap A Stack 17 Wheel B Board B5 is permanently dead due to a LV issue. This has 0% occupancy.
29 Jun 2009 10:10 Dominick Olivito Endcap A Stack 24 Wheel A Boards A21 and A22 are permanently dead due to a LV issue. These have 0% occupancy.
29 Jun 2009 10:10 Dominick Olivito Endcap A Stack 32 Wheel A Boards A21 and A22 are permanently dead due to a LV issue. These have 0% occupancy.
29 Jun 2009 10:11 Dominick Olivito Endcap C Stack 18 Wheel A Board A12 is permanently dead due to a clock line issue. This has 0% occupancy.
29 Jun 2009 10:12 Dominick Olivito Endcap C Stack 25 Wheel A Boards A21 and A22 are permanently dead due to a LV issue. These have 0% occupancy.
29 Jun 2009 10:15 Dominick Olivito Endcap A Stack 16 Wheel A Board A52 is permanently dead. This has 0% occupancy.
19 Sep 2009 10:45 Oleg Bulekov Endcap A Stack 30 Wheel A Board A51 is permanently dead. It has 0% occupancy.
24 Feb 2010 13:43 Dominick Olivito rc::ApplicationWarnings We sometimes see warnings like “Application X on host Y died while exiting. Signal 9.” at the end of a run when the ATLAS partition is terminating. Those are known and can be ignored.
06 Jun 2010 15:40 Dominick Olivito Calibration Scans Please do not call the TRT DAQ phone for problems with calibration scans on nights or weekends. Just note down the problem in the e-log and we’ll investigate. During weekdays, you can call the TRT DAQ phone and/or make an e-log entry.
05 Sep 2010 14:18 Dominick Olivito Barrel A Stack 4 There’s a misbehaving chip in Barrel A Stack 4, on board 1L. In some runs, this chip has 100% LL and HL occupancy. This is known and can be ignored for now.
22 Sep 2010 18:34 Ryan Reece CTP Busy Monitor Being 1.15% busy is the normal operating state of the TRT since we started automatically polling the front end during the beam gap, and should not be cause for alarm. This does not effect any bunch crossings with beam.
06 Oct 2010 16:33 Dominick Olivito Barrel A Stack 8 Board 3S2 (triangular section in TRTViewer) is occasionally showing lower occupancy due to LV fluctuations.
11 Nov 2010 07:51 Peter Wagner Endcap C Stack 29, Boards A31/32 occasionally show low occupancy due to LV issue.
02 Mar 2011 11:30 Jonathan Stahlman Endcap C Stack 14 Wheel A Board A41 has chip with noisy HT (seen in HT/LL ratio)
14 Mar 2011 17:35 Dominick Olivito Endcap A Stack 25 Wheel B Board B1 suffers from HV discharge issues. It will generally have higher occupancy than other boards, and in individual events may have a large number of low threshold and high threshold hits. We're currently just going to monitor this board and not take any action.
18 Mar 2011 16:55 Jonathan Stahlman Barrel C Stack 18 Board 3ML2 has a small instability in the voltage supplied to the front end. This causes the low level occupancy to fluctuate slowly over time. This issue is not urgent, but the shifter should inform the DAQ experts if the occupancy is significantly off for this board shortly prior to or during stable beams data taking.
16 May 2011 13:26 Jonathan Stahlman MRS Monitor If you see an error of the type “ERROR TRTEndcapA_E-03 TRT::CrateMonitor TTC: 0x370314: Problems during Channel Delay Scan. Some RODs may have corrupted data or go busy”, please issue a manual resync from the TRTRecoveryPanel. If the error appears again after doing the resync, please call the DAQ on call phone immediately.
27 Jun 2011 15:51 Steffen Schaepe Endcap C Stack 22 WB5 One board shows a lowered occupancy most likely due to the loss of one bridge HV contact at this place.

DAQ Instructions for Shifters

The following problems are either known or not serious and DO NOT require calling a DAQ expert:

  • When the ATLAS clock is changed, usually the following sequence occurs. This is normal:
    • an INFORMATION level message saying: “TRT would like to resynchronize its hardware" and a WARNING about "Holding trigger for TRT resync"
    • the TRT resynchronization procedure will fix any problems automatically. See the resynchronization instructions for more info.
    • note that the clock may be switched automatically at the start of a run, depending on the LHC Beam Mode!
  • Warning messages during configuration that say “More than 10 DTMROCs didn’t configure correctly.” Unless a large number (more than 10) of these messages appear at once, or a ROD goes busy at the start of a run, this isn’t serious.
  • A small number of INFORMATION level messages containing “rocketio problems” during a run (especially after an LHC ramp) are not serious. These should be fixed by the resynchronization procedure automatically.
  • A burst of messages with Message ID “ROS::” are usually not serious unless accompanied by a Busy or they come continuously. FOR MORE INFO: please read below.
  • If RODs go Busy, please try to execute the ROD Recovery procedure. If the RODs cannot be recovered by this procedure, call the TRT DAQ On Call Expert.

The following are symptoms of readout synchronization problems. If you see any of these, please execute the resynchronization procedure. If this doesn’t appear to fix the problem, then please call the TRT DAQ On-Call Expert.

  • a large number of bytestream errors seen in the online monitoring (BS errors vs lumiblock plots). Here large means more than 2% for longer than 3 lumiblocks.
  • ROS warnings coming over a period of several minutes (or more) about “timeout in request for fragment”

The following problems could be serious and may require expert intervention:

  • If a ROD goes Busy and there is no popup window at the Run Control desk to continue the run
  • If a ROD cannot be recovered by the ROD recovery procedure
  • If problems occur with the TRT resynchronization procedure, or WARNING/ERROR messages appear about getting close to the maximum resync limit

Monitoring

Date Author Comment
01 Oct 2009 Jahred Adelman Which monitoring jobs are running at the moment? Which references are we using at the moment?
13 Jul 2011 Adrian Vogel In case of problems, look at the “Troubleshooting” section of the TRT DQ Instructions.
15 Aug 2011 Adrian Vogel Applications named “DQAgent”, “Gatherer”, “Histogramming”, or “MDA” are part of the monitoring. Please contact the monitoring experts in case of questions or problems.

Active Gas System

Date Author Comment
02 Oct 2009 04:05 Zbyszek Hajduk When values of the parameters in the gas system remain very stable they are not updated. However the plot color shows that values are valid, which means that they are real.

Cooling and Temperature

Date Author Location Comment
22 Jan 2010 16:11 Ken McFarlane Barrel CO₂ Ventilation Deactivated sensors in Barrel C: CO2_S9C, CO2_S29C, CO2_S31C
Deactivated sensors in Barrel A: CO2_S2A, CO2_S16A, CO2_S18A, CO2_S32A

DCS and Alarms

Date Author Location Comment
23 Aug 2010 11:56 Jim Degenhardt Alarm Panel “TRT ATLTRTLCSX ArchiveBuffer Number Error.” This is a symptom of an ATLAS-wide problem. Check with the ATLAS DCS expert before calling the TRT DCS expert. Solution seems to be restart of RDB manager for sub-system after invention on ATLAS central system.

HWI

Date Author Location Comment

Recommended DAQ Panel Settings

The tdaq-04-00-01 is taken automatically from the central twiki and is outdated...

Use tdaq-05-05-00 instead of tdaq-04-00-01.

Version tdaq-04-00-01
Setup Script /det/tdaq/scripts/setup_TDAQ_tdaq-05-05-00.sh
Partition Name ATLAS
Database File /atlas/oks/tdaq-05-05-00/combined/partitions/ATLAS.data.xml (ViewVC)
MRS Filter TRT
OHP Opt -c /atlas/moncfg/tdaq-05-05-00/indet/ohp/config.xml
TriP Opt -c /atlas/moncfg/tdaq-05-04-00/indet/trp/config.xml

Phone Numbers

System Numbers Who/Where/What
Control Rooms 71343
70946
71355 		ACR TRT Desk (3162-R-K01)
ID SCR (3159-R-008)
DQ SCR (3162-2-C01)
Run Coordinators 160547
168175
160772
160412 		On-Call Phone
Andrea Bocci
Dominik Derendarz
Anatoli Romaniouk
TRT DAQ 160531
167106 		On-Call Phone
Chris Meyer
TRT DCS 160242 On-Call Phone
TRT Monitoring 160543
On-Call Phone
TRT High Voltage 160412 Anatoli Romaniouk
TRT Active Gas 167309
On-Call Phone
ID General 162449 On-Call Phone
Other Systems   see the ATLAS Phone List

When in doubt, always call the TRT Run Coordinators’ On-Call Phone. If you are experiencing a specific problem, you may call the system expert directly.

Reaching CERN Phones from the Outside
Fixed +41 22 76 xxxxx  
Mobile +41 75 411 xxxx (omit the leading 16)
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng bijan_logmanager.png r1 manage 165.7 K 2014-07-09 - 07:39 UnknownUser  
Edit | Attach | Watch | Print version | History: r494 < r493 < r492 < r491 < r490 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r494 - 2018-08-31 - kzhukov_40CERN_2eCH
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding ATLAS?Please contact the page author (see Topic revision above) or the Run Coordinator of the specific system.
Contact SysAdmins support only for technical issues