setup VM

• na to maju tiez nejake skripty + ansible
• https://github.com/galaxyproject/admin-training/tree/2023-gent/bootstrap-instances
• aj training
• DNS veci ale asi uz musia byt nastavene

Installing Galaxy with Ansible

• info o systeme kde pustim playbook

ansible -i hosts -m setup my_hosts | less

• good practive to close jinja formatting withing double quotes (because otherwise yaml parser can have issues)
• ansible is not great about undoing changes
• its okay to make a copy of a role and make local changes (e.g. usegalaxy.eu does that too)
• galaxy-ansible role of certain version can only install Galaxy up to a certain version
• ak pocas prvej instalacie sa mi zosype terminal (napr. broken pipe), tak sa Galaxy a vsetky potrebne services nespustia spravne !
• prve prihlasenie, resp. registracia vytvori admin usera
• requests certificate failed sa moze stat - treba zbehnut znova
• usegalaxy.cz vo vyvoji
  • https://github.com/CESNET/usegalaxy

Server Maintenance

• toto by mohlo byt pouzite pri vytvoreni noveho Galaxy a zaroven zachovanie DB
  • https://training.galaxyproject.org/training-material/topics/admin/tutorials/backup-cleanup/tutorial.html#restoration

Customization

• je mozne menit logo, nadpis, farby a hlavne uvodnu html stranku
• doporucene pouzit iFrame

TUS

• umoznuje bezat upload async mimo samotneho Galaxy
• command systemctl status tusd-main nefunguje
  • outdated - treba urobit sudo systemctl start galaxy-tusd

Data Managers

• toto by slo mozno vyuzit na napr. storage velkych DB, ktore sa pouzivaju v tooloch

CVMFS

• Cern virtual machine file system
• sharing huge datasets
• it uses AutoFS, downloads files and repositories only on-demand, cache available

Apptainer

• umoznuje pouzivat singularity containery
• Singularity is an open source container platform designed to be simple, fast, and secure. Unlike Docker containers which requires root privileges to run containers, Singularity is designed for ease-of-use on shared multiuser systems and in high performance computing (HPC) environments.
• v admin -> manage dependencies mozeme najst info o stiahnutych containeroch
• novy paper o planemo: The Planemo toolkit for developing, deploying, and executing scientific data analyses in Galaxy and beyond

Ephemeris

• ephemeris poskytuje vela uzitovnych toolov
• mozeme instalovat tools na nasej instancii pomocou shed-tools
  • shed-tools install -g https://galaxy.example.org -a <api-key> -t workflow_tools.yml
• k tomu pouzit napr. list toolov extrahovanych z workflow
  • workflow-to-tools -w mapping.ga -o workflow_tools.yml -l Mapping
• tiez umoznuje testovat tooly - tu je vlastne moznost spustit realne testy specifikovane v tool configu
  • toto vytvori test_history na danom Galaxy
• ziskanie vsetkych toolov na nejakej instancii
  • get-tool-list -g "https://umsa.cerit-sc.cz/" -o "umsa_files.yaml"
  • to by slo vyuzit pri migracii
• neexistuje uninstall https://github.com/galaxyproject/ephemeris/issues/83
• usegalaxy.eu ma napr. list toolov, cize clovek moze pull requestom poziadat o automaticku instalaciu

Data Libraries

• umoznuje pouzitie data libraries
  • Before we can import local data, we need to configure Galaxy to permit this
• Datasets in libraries do not count against user quotas
• ephemeris moze byt pouzite na vytvaranie libraries (setup-data-libraries)

Scripting Galaxy with BioBlend

• idea - galaxy sa da pouzivat ako fancy vypocetny server, ktoremu posielam vypoctove ulohy zo skriptov
• novsie Galaxy maju automaticky pristupnu dokumentaciu API na suffixe api/docs
• bioblend poskytuje API wrapper
  • vytvaranie, editovanie, mazanie historii
• prekryv medzi BioBlend a Ephemeris?
  • jedno je command line tool a druhe python library

User, Groups, and Quotas management

• expose_dataset_path - Users to see the full path of datasets via the "View Details" option in the history
• quotas should be definitely set for anonymous users !!!
• galaxyeu ma system, kde na poziadanie sa cloveku zvysi quota

Cluster

• there is also user limit - maybe that's why we often wait for some jobs on UMSA?
• even running Slurm on the same machine is useful, because it allows to e.g. restart Galaxy without interrupting the jobs
• destinations were renamed to environments
  • which plugin (slurm/pulsar/...)
  • on docker? what container?
  • queue, cores, memory
  • env variables
• handlers/runners have specified workers - number of jobs that can run at the same time using the same handler
• job plugins usually require shared filesystem (except Pulsar)
• dynamic job runner
  • total perspective vortex or arbitrary python function to choose appropriate environment for a job
• Slurm
  • config generator
  • v logu je uzitocne si vsimnut:
    • submitting file /srv/galaxy/server/database/jobs/000/2/galaxy_2.sh: This is the path to the script that is submitted to Slurm
    • (1) queued as 4: Galaxy job id 1 is Slurm job id 4
    • we can see then details using scontrol show job 4If job id ended is reached, the job should show as done in the UI
• Job Metrics
  • e.g. what is max memory my job used?
  • can be gathered using galaxy_job_metrics_plugins, we can specify what we want to collect

Job Destinations

• allows to control resources we assign to a job
• k tomu je uzitocne pouzit TPV
• DB toolov a ich vhodnych nastaveni resourcov - chceme tam dat nase?
• Configuring TPV to process resource parameters
  • aby sme pre tool testing umoznili zadanie resources ako parameter
  • treba este pridat do group_vars/galaxyservers.yml pod galaxy_job_config - tools toto:

    - id: testing
      environment: tpv_dispatcher
      resources: testing

• kde id je ID toolu, do ktoreho chceme pridat takuto moznost - i ked v realite to asi nie je take jednoduche
  • mozno by to slo nastavit v default: []
  • co sa teda stane, ak zmazem tool? co ak ho este nemam?

Pulsar

• umoznuje spustat joby na strojoch, ktore nezdielaju suborovy system
• na komunikaciu sa pouziva RabbitMQ
• After running the playbook, after RabbitMQ restarts:
  • On the oz/Pulsar VM: sudo systemctl restart pulsar
  • On the eu/Galaxy VM: sudo galaxyctl restart

Celery

• can do some async IO tasks processing
• useful for huge Galaxy servers with high load

Reports

• can provide overview of jobs and users etc, but not secured, can break GDPR etc

gxadmin

• gxadmin query jobs (ako galaxy user)
  • zoznam jobov, ktory handler, destination, user, ID
• gxadmin report job-info <ID>
  • napr. Tool Parameters
• gxadmin query latest-users
  • when joined, disk usage
• gxadmin query queue-overview
  • mozeme pozorovat ake joby akurat bezia alebo cakaju
  • da sa pouzit na generovanie grafov https://stats.galaxyproject.eu/

Telegraf and Grafana

• galaxy_url/grafana
• je celkom jednoduche rozbehat Grafanu na svojom serveri a potom importovat JSON config napr. z galaxyeu
• kedze vsetko je (v teorii) nakonfigurovane rovnako, tak to funguje
• je mozne si aj nastavit notifikacie, ked napr. hodnota niecoho sa dostane nad prahovu hodnotu

Sentry

• tutorial nefunguje, treba este
  • pridat sentry_domain: "gat.\{\{ groups['galaxyservers'][0] \}\}" do group_vars/all.yml
  • In group_vars/secrets.yml, adjust the sentry URL from https:// to http://
  • a v nejakom random momente spustit ako root:

cd /srv/sentry
docker-compose down -v
docker volume rm sentry-clickhouse sentry-data sentry-kafka sentry-postgres sentry-redis sentry-symbolicator sentry-zookeeper
rm -rf /srv/sentry

• ale inak tool moze automaticky zbierat errory jednak na Galaxy urovni a Pulsar serverov, ale aj z toolov
  • a prezentuje ich v peknej a prehladnej forme (aka. netreba sa hrabat v logoch)
  • mozeme odhalit error este predtym, ako si ho user vsimne
• ked sa error opakuje, tak sa zgroupnu dokopy

Galaxy troubleshooting

• mala by byt nahravka tejto prezentacie
  • https://psu.mediaspace.kaltura.com/media/Galaxy+Admin+Training+2023+Troubleshooting+Session/1_ouk96yh4
  • Command Line Records - https://asciinema.org/a/579121
• start with logs - there are many of them! (many sub-services)
  • go to /var/log and run ls -lrt to see recently updated logs
• e.g. startup issues
  • database migration - should be covered by ansible, but...
  • stuck in restart loop
    • go to end of log using journalctl -e -u 'galaxy-*' and see log just before last start attempt
• atop - ako htop ale detailnejsie a uklada si historiu?
• load average v htop - should be lower than number of cores
• profiling
  • py-spy dump/top --pid <process PID> (ako root)
  • ps -axf | grep galaxy
• in /srv/galaxy/var/configs we have bunch of configs related to tools
  • in /srv/galaxy/var/shed_tools installed tools
  • je fajn si nastavit, aby defaultne joby, ktory failnu sa nezmazali ich logy
    • cleanup_job to onsuccess (kde?)
  • srun --pty bash - spusti terminal na VM kde ma job bezat (aj v spravnom env?)
    • potom sbatch galaxy_<job ID>.sh (slurm)
• "gray" queued jobs
  • depends on job handler configuration
  • gxadmin query jobs --nonterminal