The monitoring tool
veperf was introduced in November 2018 in the
blog post VE monitoring with
is part of the py-veosinfo
package and displays live performance metrics of own running VE
veperf has been extended to display some additional metrics: load
and store memory bandwidths, power consumption of each monitored VE
and energy used by each VE since the monitoring tool was started.
In order to monitor memory bandwidths the VE processes need to be started with the environment variable setting:
This is valid for programs linked with any of the NEC SDK compilers: ncc, nc++, nfort. Programs linked with LLVM are currently not influenced by the environment variable and will not display memory bandwidth metrics.
py-veosinfo source package is uploaded to pypi.org and can be installed
pip install py-veosinfo
The build needs the RPM
veosinfo-devel to be installed as it needs
access to the include file
For local user installs the binary is being created in
For CentOS 7 (Python 2.7) setups we provide an rpm in the YUM repository with the following configuration:
$ cat /etc/yum.repos.d/ve_extras.repo [ve-extra] name=Aurora TSUBASA Extras baseurl=https://sx-aurora.com/repos/veos/ef_extra gpgcheck=0 enabled=1
With that configured the installation command is
sudo yum install py-veosinfo
The executable ends up installed in
Local install without root privileges
If you have no root privileges you can unpack the localinstall tarball in your home directory:
cd ~ wget https://github.com/SX-Aurora/py-veosinfo/releases/download/v2.5.1/py-veosinfo-2.5.1-localinstall-py27-py36.tgz tar xzvf py-veosinfo-2.5.1-localinstall-py27-py36.tgz
It is essential to do this in
$HOME because that puts the veosinfo
(binary) module in a default place where python finds it without need
to manipulate PYTHONPATH.
Normally you should be able to now execute
~/.local/bin is being added to your
$PATH if it exists, on RHEL or
veperf shows performance metrics of the user’s own processes on all or selected nodes, measured over certain time intervals. The root user can monitor all VE processes’ performance metrics. Performance counter readout is done from the vector host (VH) with no overhead or interruption for the running processes, no instrumentation of programs is necessary. It involves reading register values for registers that don’t need the stopping of the cores and uses the VE’s system (privileged) DMA facilities.
The tool can be used by administrators to track the overall performance of VEs and detect VEs running at very low performance. Detecting “slow” users is intentionally left as a separate step. Also the tool can be started by users just before running a job and gather the performance evolution of their program over the run-time. It can help identify fast and slow phases of the program.
veperf [-h|--help] [-d|--delay DELAY] [-n|--node NODE] [INTERVALS]
-d|--delay DELAY: performance metrics measurement time interval. Default: 2s.
-n|--node NODE: select VE node ID for measurement. Multiple –node options are allowed. When the option is omitted, all nodes are monitored.
INTERVALS: number of monitoring time intervals before exiting the program. When this option is not specified, ‘veperf’ will continue until it is killed.
The output for each monitoring time interval consists of:
- a line containing the labels of the thread monitoring data
- for each monitored VE node:
->VE#: the VE node ID
- for each (own, seen) task running on the VE
- pid, USRSEC, EFFTIME, MOPS, MFLOPS, AVGVL, VOPRAT, VTIMERATIO, L1CMISS, PORTCONF, VLLCHIT, LOADBW, STOREBW, comm
- summed up values for MOPS, MFLOPS, POWER and ENERGY for the VE. If processes are set to monitor memory bandwidths, the summed LOAD and STORE memory bandwidths (Core <-> LLC) are displayed.
Only the visible processes contribute to the MOPS, MFLOPS, LOADBW and STOREBW sums while POWER and ENERGY might include contributions from processes that are not visible, belonging to other users.
The example below shows 8 processes running on VE node 1 with an idle node 0, monitored with the default time interval of 2s.
pid USRSEC EFFTIME MOPS MFLOPS AVGVL VOPRAT VTIMERATIO L1CMISS PORTCONF VLLCHIT LOADBW STOREBW comm -> VE0 SUM VE0: MOPS=0 MFLOPS=0 POWER=40.6W ENERGY=4.23kJ -> VE1 199287 301.85s 1.000 75934 67531 175 98.6% 91.0% 0% 0% 96% 36.091 23.460 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199288 301.84s 1.000 76016 67630 176 98.6% 91.1% 0% 0% 97% 35.963 23.457 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199289 301.82s 1.000 76170 67759 175 98.6% 91.2% 0% 0% 97% 36.109 23.493 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199290 301.88s 1.000 75960 67581 176 98.6% 91.3% 0% 0% 97% 35.938 23.459 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199291 301.82s 1.000 75664 67277 175 98.6% 91.1% 0% 0% 96% 35.985 23.445 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199293 301.87s 1.000 75368 67017 176 98.6% 90.8% 0% 0% 96% 35.791 23.378 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199295 301.85s 1.000 75839 67449 175 98.6% 91.0% 0% 0% 96% 35.996 23.456 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199298 301.81s 1.000 75797 67420 176 98.6% 91.2% 0% 0% 97% 35.919 23.446 cpunodebind=-1--/scr/qe-6.3/bin/pw.x SUM VE1: MOPS=606749 MFLOPS=539664 LOADBW=287.791GB/s STOREBW=187.593GB/s POWER=143.4W ENERGY=15.31kJ pid USRSEC EFFTIME MOPS MFLOPS AVGVL VOPRAT VTIMERATIO L1CMISS PORTCONF VLLCHIT LOADBW STOREBW comm -> VE0 SUM VE0: MOPS=0 MFLOPS=0 POWER=40.6W ENERGY=4.31kJ -> VE1 199287 303.85s 1.000 58922 50487 174 98.4% 89.6% 0% 0% 96% 35.156 24.599 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199288 303.84s 1.000 58653 50276 175 98.4% 89.9% 0% 0% 96% 34.786 24.543 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199289 303.83s 1.000 59104 50677 174 98.4% 90.0% 0% 0% 96% 35.095 24.591 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199290 303.88s 1.000 58598 50235 175 98.4% 89.7% 0% 0% 96% 34.740 24.515 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199291 303.82s 1.000 58731 50330 174 98.4% 89.8% 0% 0% 96% 34.949 24.547 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199293 303.87s 1.000 58166 49826 175 98.4% 89.3% 0% 0% 96% 34.605 24.444 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199295 303.85s 1.000 58808 50405 174 98.4% 89.4% 0% 0% 96% 34.947 24.561 cpunodebind=-1--/scr/qe-6.3/bin/pw.x 199298 303.81s 1.000 58540 50175 175 98.4% 89.7% 0% 0% 96% 34.737 24.518 cpunodebind=-1--/scr/qe-6.3/bin/pw.x SUM VE1: MOPS=469522 MFLOPS=402411 LOADBW=279.014GB/s STOREBW=196.318GB/s POWER=152.8W ENERGY=15.62kJ
- USRSEC: Task’s user time on VE. When the task is not scheduled on a core, this time does not progress.
- EFFTIME: Effective time: ratio between user and elapsed time. A value lower than 1.0 is a sign that the task is spending time in syscalls.
- MOPS: Millions of Operations Per Second.
- MFLOPS: Millions of Floating Point Ops Per Second.
- AVGVL: Average vector length.
- VOPRAT: Vector operation ratio [percent].
- VTIMERATIO: Vector time ratio (vector time / user time) [percent].
- L1CMISS: SPU L1 cache miss time [percent].
- PORTCONF: CPU port conflict time [percent].
- VLLCHIT: Vector load LLC hits [percent] (counter not active with MPI).
- LOADBW: LLC to VE core load memory bandwidth in GB/s. Only shown if
- STOREBW: LLC to VE core store memory bandwidth in GB/s. Only shown if
- POWER: Power consumption of the VE in Watts. No VH related power consumption is considered.
- ENERGY: Energy consumed by the VE since
veperfwas started, in kiloJoules. No VH related power consumption is considered.
NOTE: All metrics except USRSEC, POWER and ENERGY are giving the values over the last displayed time interval.
USRSEC shows the time that the executable was using the VE core. POWER is measured at the end of the time interval!
If you care about an accurate power measurement, chose a small time interval with the
-d option, 1 or 2s. ENERGY is
a time integral of the power from the start of the execution of the
veperf tool, thus if started right before the program and stopped
right after the program has finished, you’ll get the energy spent by each VE in the last displayed output block.
The PMMR register of each VE core is controlling which performance
metrics are actually active. Currently there is no easy way to control
this register from user side besides setting
VE_PERF_MODE, MPI and
non-MPI programs might measure slightly different metrics.