TGC FE expert guide

Another experts' instruction here

How to recover from a power cut

• See step-by-step instruction here : muon_2012_11_28.pdf:

CCI/HSC crates power supply control and monitoring

• Reference manual pdf, ppt

PS Reset / DCS Reset : ( EXPERT ONLY )

• Do not perform the following operation if you do not understand what you are doing
• PS Board Reset
  1. login to SBC for TTC crate
     • A-Side: sbc-tgc-tcc-eca-01.cern.ch
     • C-Side: sbc-tgc-tcc-ecc-01.cern.ch
  2. source the setup script for TGC Frontend environment

     source /det/muon/TGCFE/installed/bin/TGC_env.sh

  3. run the following command

     TgcTtcvi -c !PsReset 

  4. type yes if you are sure to send the signal.

• DCS Reset
  1. loguin to SBC and source the setup script (same as PsReset)
  2. run the following command

     TgcTtcvi -c DcsReset 

  3. type yes if you are sure to send the signal.

open TGC ROD Status panel

• You can open TGC Rod status panel alone without TDAQ panel.
  • First, you long in to ATLAS point 1 computer via "atlasgw" from your computer.

     ssh -Y atlasgw 

  • pc-tgc-daq-eca-01 or pc-tgc-daq-ecc-01 is one of good computers to login for this purpose.
  • You source a set-up file and run a command.

     source  /det/muon/TGC/scripts/setupGlobal.sh 
     TGCpanel 

  • Thne, the panel will appear.

Reboot all cci sbcs

• procedure
  1. login to a machine in Point1. (The script includes SSH command inside it.) and run followings:

     souce /det/muon/TGCFE/installed/bin/TGC_env.sh
     oncall_allccisbc_reboot.sh

• You will be asked twice and please type "Y" (capital y) to confirm to reboot the SBCs.

Reset Semaphore

• procedure
  1. login to a machine in Point1. (The script includes SSH command inside it.) and run followings:

     souce /det/muon/TGCFE/installed/bin/TGC_env.sh
     oncall_allsbc_resetSemaphore.sh

• In case "semaphore" is locked, you will see following messages
  

How to find ERROR LOG ?? (for further investigation of problems reported in MRS)

• location of log-file ... atlasgw:/clients/scratch/logs/tdaq-04-00-01/ATLAS/

  side & crate	file name
  A-Side, TTC	TGCTTC-A_RCD_PhysicsMode_sbc-tgc-tcc-eca-01.cern.ch_[UTC].out/err
  C-Side, TTC	TGCTTC-C_RCD_PhysicsMode_sbc-tgc-tcc-ecc-01.cern.ch_[UTC].out/err
  A-Side, FE	TGCFE-RCD_ECA[01-04]_Physics_sbc-tgc-cci-eca-[01-04].cern.ch_[UTC].out/err
  C-Side, FE	TGCFE-RCD_ECC[01-04]_Physics_sbc-tgc-cci-ecc-[01-04].cern.ch_[UTC].out/err
  A-Side, SL	TGCSL_ReadoutApplication-CSL[04-06]_sbc-tgc-csl-lvl1-[04-06].cern.ch_[UTC].out/err
  C-Side, SL	TGCSL_ReadoutApplication-CSL[01-03]_sbc-tgc-csl-lvl1-[01-03].cern.ch_[UTC].out/err

Analyze Calibration DATA and update parameter and OKS : 3-STEPS

step1: Analysis of Calibration data

1. preparation : login, go to working space, set environment (the computer OUTSIDE of Point1.)

   ssh -X pc-tgc-calsrv-01.cern.ch
   cd /det/tgc/Monitoring/hist 
    source   /det/muon/TGCFE/installed/bin/TGC_env.sh

2. Then, get rod files from Point 1 … * Get files from Point 1 (replace "runumber")

   sh OncallTool_GetFilesFromPoint1.sh RandomRunNo TrackRunNo ASDRunNo

   or

   sh OncallTool_GetFilesFromPoint1_single.sh RunNo

   Copy data with SCP, your passswd @P1 is required. * Check if TGC shifter copied data to shared disk
   • • If data transfer has been completed by shifter, it is found under /det/tgc/data/runs/
     • If it is not the case, you should copy them by yourself with following procedure.
3. Decode data

   do_Decoding_pararel.sh runnumber 100000 
   do_Decoding_pararel.sh runnumber -1

   Random : decode 100k events. ASD and Track : decode all events.
4. Run analysis program.
   • Case : ASD
     1. run the analysis code and check the result.

        sh OncallTool_MissingChannelCheck.sh runnumber

        Check the number and comper them to the reference in parenthesis "()"
     2. If it looks strange, check histogram with

        sh setup.sh 

        (will open familiar GUI tool)
     3. Finally send it to USA 15 in Point1. Following instruction will be shown with appropriate parameter so that you can just copy & paset
        1. transfer the missing channels list to computer in the Point1 as below.

           scp /det/tgc/Monitoring/hist/missinglist/run${runnum}_missing.txt  atlasgw:/shared/data/crmuon/

        2. after that, login to atlasgw, and do this.

           ssh atlasgw
           mv /shared/data/crmuon/run${runnum}_missing.txt  /det/tgc/Monitoring/hist/missinglist

   • Case : Random
     1. run the analysis code and check the result.

        sh OncallTool_NoisyChannelList.sh   runnumber

     2. If noisy channel(s) was found, prepare configuration parameter. Following instruction will be shown with appropriate parameter so that you can just copy & paset
        1. check which channel is found a new noisy channel. EXCEPTION It is known that we cannot kill 1 single noisy channel

           C       05      0       E       W       2       2       41 

           . In case only this channel is listed, it is not needed to update parameters.
        2. send noisy channel list to point 1.

           scp /det/tgc/Monitoring/hist/noisylist/run${runnum}_bcidmasklist.txt atlasgw:/shared/data/crmuon/

        3. Log in to Point 1 machine.

            ssh atlasgw 
            cp    /shared/data/crmuon/run${runnum}_bcidmasklist.txt  /det/muon/TGCFE/src/tgc_hw/scripts/noisy_channels/

        4. run a update command. I recommend to use "screen" command before do followings.

            screen  
            cd /det/tgc/FEtest/scripts/expert/ ; sh update_psb_parameters.sh

           The program will ask you to check the new parameter showing the parameter contents. Please compare it to last on-call shift summary and confirmed the difference is reasonable.
           Did you miss-type "yes" ? You don't have to wait for preparing parameters again. try below.

            (this is a command to restart the process.) $ cd /det/tgc/FEtest/scripts/expert/ ; sh put_psb_DetMuonTgcfe.sh

        5. After it finishes, upload parameters to OKS. see "Upload parameters (text-base) to ORACLE" in this page.
           If you found new missing channels, please do follwoing two things before updating missinglist in P1.
        • v-threhold up/down operation.
          1. Set value for a_direction as "+" insted of "to" ,then set value for s_val as 0.01.
          2. Set value for s_direction as "- " insted of "to" ,then set value for s_val as 0.01.
        • If the situation doesn't improve, please check the hitprofile to figure out a position of the problem. And please consider how to access to it during a technical stop, long shutdown,,, etc.
   • Case : Track
     1. Open a gui tool.

        sh setup.sh

     2. do "Trigger/Readout cor" with "all BW" option.
     3. Histogram will be produces in the directory "psfile". Check it.
5. Finally, report the results.

step 2: Upload parameters (text-base) to ORACLE

generate new parameter files according to the result of the test during the daily calibration hour (see below).

1. Login to the TGC machine;

   pc-tgc-daq-eca-01, pc-tgc-daq-ecc-01 or pc-tgc-mon-rod-01 

2. Source the setup script (if not yet)

   /det/muon/TGCFE/installed/bin/TGC_env.sh

3. Move to the following directory;

   cd /det/muon/TGCFE/installed/bin/oracle/

4. Run a wrapper script;

   auto_upload.sh 

5. Type yes and enter # of trial (if you don't try to upload yet, please input 0 as the number)
6. Then, the following processes will be carried out;
   1. start reading text file (stored in /det/muon/TGCFE/config/Frontend/)
   2. uploading parameters to ORACLE
   3. compare parameters in between text file and ORACLE
7. rule of DB tag (generated automatically):
   • Physics_[date]_normal_[# of trial] (3 out-of 4)
   • Physics_[date]_l67_[# of trial] (2 out-of 2)
   • Example: Physics_100301_normal_0, Physics_100301_l67_0
8. Finally, the result of the comparison is shown in the terminal. If there is no inconsistent parameter, please post the message to e-log.
9. After it finishes, upload parameters to OKS. see "Change OKS xml files for the global partition" in this page.

step 3: Change OKS xml files for the global partition

Do not perform the following operation if you do not understand what you are doing

1. source the setup script (if not yet)

   /det/muon/TGCFE/installed/bin/TGC_env.sh

2. move to the following directory and source the script;

   cd /det/tgc/db/
   source mycheckout.sh

3. Then, the oks_dsata_editor GUI pannel will be appeared in your terminal automatically.
4. Please change the attribute which you want.
   • Example: change Oracle DB tag for Frontend configuration
     1. search TGCFE_DBSetting class in the list (enter the class name in "Matching names of classes" box)
     2. double click the class and open it (new window will be shown)
     3. sort the list by FE_Trigger (or FE_Mask or FE_Config), and then you can find some objects with "Physics_xxxx_yyyy_zzzz".
     4. double click these objects, and right click the following attribute to change the DB tag;
        • FE_Trigger
        • FE_Mask
        • FE_Config
        • (Please note that you should change all of them one-by-one. More easy way will be implemented soon.)
     5. check the all attributes in the list are correct or not.
5. Please execute "FIle --> Save All Updated" to save all changes.
   • PLEASE input some comments to be able to identify the modification.
6. Dump the TGCFE parametars by useing local xml file as a test.

   dumpTgcFEParameters -f /det/tgc/db/tdaq-04-00-01

   • If correct parameter dumped, go to next step.
   • If NOT correct parametrs dumped, please do follwoing two things. DON'T commit the xml files.
     1. clean up /det/muon/TGCFE/config/FrontEnd/oncall/dump and try to perfome dumpTgcFEParameters.
     2. If you get not correct parameters again, please go back to step 2 and make parameters again.
7. Please execute "File --> Commit User Repository" to commit all updated xml files to CVS server.
   1. The same comments which you input above will be shown in the window.
   2. Please click commit button to upload files.
8. If no error message is shown, please post message to the e-log.
9. For avoiding mis-understanding, Announce Shiftleader and Run Control Shifter, what you changed,why you changed.
   • What you changed (/atlas/oks/tdaq-04-00-01/muons/segments/TGC/FE/TGCFE-A(&C)_Segment.data.xml)
   • Why you changed (updating mask parameter)

Chcek eLMB Internal mode at EIFI

When there is [1] power glitch or [2] LV power-cycle, it can be that wrong eLMB modes are configured into eLMB at EIFI. In such a case, please check internal modes of eLMB, following the instruction.

1. navigate down the FSM tree:
   • TGC -> SIDE -> DCS -> Driver -> DCS EIFI -> ELMB EIFI
   • Note that the FSM node itself is disabled. This is done on purpose and has nothing with this action.

1. compare values between "Image" and "Current".

1. If there is inconsistency found, please fixed them by left click on the name of eLMB. (the value in leftmost column)

Delay Scan

1. launching DAQ_Panel
   • TDAQ ->DAQPanel
   • ( TDAQ_Configuration )
     • Setup Script: /det/muon/TGCFE/installed/bin/setup_partTGC_DelayScan.sh
     • Part Name: partTGC_DelayScan
     • Database File: /det/muon/standalone/databases/tdaq-02-00-03/muons/partitions/part_TGC_DelayScan.data.xml
     • Setup Opt: -newgui
     • Oks Opt:
     • MRS Filter: TGC
     • OHP opt:
     • BUSY Opt:
     • OMD Opt :
     • TriP Opt:
   • Click "Read Info" button

1. executing the script for DelayScan
   • $> cd /det/muon/TGCFE/TGCFeOperation/DelayScan/script
   • $> sh DelayScan.sh
     • Delayscan will start. It will take about 30 minutes until finish.

1. transporting raw data to out of P1 machine
   • Raw data of DelayScan are put in "/det/muon/TGCFE/TGCFeOperation/DelayScan/script/rootfile/". you need to copy this files to TGC local machine (pc-tgc-calsrv-01). Please use the GUI for TgcCalibration.("https://pc-atlas-www.cern.ch/twiki/pub/Main/HowToRunTgcCalibration").

1. data analysis

• Decoding data and making timing histograms. Doing at pc-tgc-calsrv-01. *setup *sourse /det/muon/TGCFE/installed/bin/TGC_env.sh
  • data decoding
    • do_DecodingDelayscan.sh [runnum]
  • data analysis
    • $> cd /det/muon/TGCFE/TGCFeOperation/DelayScan/script
    • $> sh cable_ASDall.sh [runnum]
  • feed buck timing
    • under construction

Standalone cosmic run

Use part_TGC2. Setup partition as figures show.

TgcExpertOperationObsoletes