Software 2
New Software
New software can be installed system-wide at the request of users, provided it meets the following criteria:
- It is either freely available or NU has a site license for it.
- It is compatible with the existing OS environment on Shabyt.
- It can utilize resources available on Shabyt effectively.
For assistance regarding new software packages, please contact Shabyt system administrators at hpcadmin@nu.edu.kz.
Software Priorities
Software applications are installed in accordance with priorities.
- Software which can be installed via EasyBuild application falls under this category. List of supported EasyBuild softwares you can find here: EasyBuild
- Applications that are crucial for User Groups but cannot be installed through EasyBuild fall under this priority.
- Individual user requests. Such requests will be processed only after other priorities have been fulfilled.
Software List
| Name | Description | Versions |
|---|---|---|
| AMD-uProf | AMD uProf is a performance analysis tool for applications running on Windows, Linux & FreeBSD operating systems. It allows developers to better understand the runtime performance of their application and to identify ways to improve its performance. | AMD-uProf/3.5.671 |
| Anaconda3 | Anaconda: Python Data Science Platform for Python 3 | Anaconda3/2022.05 |
| AOCC | AOCC is a high-performance x86 CPU compiler for C, C++, and Fortran programming languages | AOCC/4.0.0 |
| archspec | Archspec provides a standard set of human-understandable labels for various aspects of a system architecture like CPU, network fabrics, etc. and APIs to detect, query and compare them. | archspec/0.1.3 |
| ATK | ATK provides the set of accessibility interfaces that are implemented by other toolkits and applications. Using the ATK interfaces, accessibility tools have full access to view and control running applications. | ATK/2.38.0 |
| Automake | Automake – make file builder part of autotools | Automake/1.16.1
Automake/1.16.3 Automake/1.16.4 Automake/1.16.5 (D) |
| Bison | Bison is a general-purpose parser generator that converts an annotated context-free grammar into a deterministic LR or generalized LR (GLR) parser employing LALR(1) parser tables. | Bison/3.5.3
Bison/3.7.1 Bison/3.7.6 Bison/3.8.2 (D) |
| cairo | Cairo is a 2D graphics library with support for multiple output devices. Currently supported output targets include the X Window System (via both Xlib and XCB), Quartz, Win32, image buffers, PostScript, PDF, and SVG file output. | cairo/1.16.0
cairo/1.17.4 (D) |
| CGAL | CGAL is an open source software project that provides easy access to efficient and reliable geometric algorithms in the form of a C++ library. | CGAL/4.14.3 |
| CMake | CMake is the standard for building C++ code | CMake/3.20.1
CMake/3.21.1 CMake/3.22.1 CMake/3.23.1 (D) |
| CUDA | CUDA (formerly Compute Unified Device Architecture) is a parallel computing platform and programming model created by NVIDIA and implemented by the graphics processing units (GPUs) that they produce. CUDA gives developers access to the virtual instruction set and memory of the parallel computational elements in CUDA GPUs. | CUDA/11.4.1
CUDA/11.7.0 (D) |
| double-conversion | Efficient binary-decimal and decimal-binary conversion routines for IEEE doubles. | double-conversion/3.2.0 |
| Doxygen | Doxygen is a widely-used documentation generator tool in software development. It automates the generation of documentation from source code comments, parsing information about classes, functions, and variables to produce output in formats like HTML and PDF. | Doxygen/1.9.1
Doxygen/1.9.4 (D) |
| Eigen | Eigen is a C++ template library for linear algebra: matrices, vectors, numerical solvers, and related algorithms. | Eigen/3.3.9
Eigen/3.4.0 (D) |
| ELPA | Eigenvalue SoLvers for Petaflop-Applications. | ELPA/2021.11.001 |
| expat | An XML parser library written in C. | expat/2.2.9 |
| FFTW | FFTW is a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, i.e. the discrete cosine/sine transforms or DCT/DST) | FFTW/3.3.9
FFTW/3.3.10 (D) |
| FFmpeg | FFmpeg is the multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play audio stream and video. | FFmpeg/4.3.2
FFmpeg/4.4.2 (D) |
| FLAC | FLAC is an audio format similar to MP3, but lossless, meaning that audio is compressed in FLAC without any loss in quality. | FLAC/1.3.4 |
| flex | Flex (Fast Lexical Analyzer) is a tool for generating scanners. A scanner, sometimes called a tokenizer, is a program which recognizes lexical patterns in text. | flex/2.6.4 |
| freetype | FreeType 2 is a software font engine that is designed to be small, efficient, highly customizable, and portable while capable of producing high-quality output (glyph images). | freetype/2.11.0
freetype/2.12.1 (D) |
| GCC | The GNU Compiler Collection (GCC) provides C/C++ and Fortran compilers. | GCC/9.3.0
GCC/9.4.0 GCC/10.3.0 GCC/11.2.0 GCC/11.3.0 GCC/12.2.0 (D) |
| GDAL | GDAL is a translator library for raster and vector geospatial data formats that is released under an MIT style Open Source License by the Open Source Geospatial Foundation. As a library, it presents a single raster abstract data model and single vector abstract data model to the calling application for all supported formats. It also comes with a variety of useful command line utilities for data translation and processing. | GDAL/3.5.0-foss-2022a |
| GEOS | GEOS is a C/C++ library for computational geometry with a focus on algorithms used in geographic information systems (GIS) software. | GEOS/3.10.3 |
| GLPK | The GLPK (GNU Linear Programming Kit) package is intended for solving large-scale linear programming (LP), mixed integer programming (MIP) problems. | GLPK/5.0 |
| GMP | GMP is a free library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating-point numbers. | GMP/6.2.1 |
| GObject-Introspection | GObject introspection is a middleware layer between C libraries (using GObject) and language bindings. The C library can be scanned at compile time and generate metadata files, in addition to the actual native C library. | GObject-Introspection/1.68.0
GObject-Introspection/1.72.0 (D) |
| GSL | The GNU Scientific Library (GSL) is a numerical library for C and C++ programmers. It is free software under the GNU General Public License. | GSL/2.7 |
| GTK2 | The GTK+ 2 package contains libraries used for creating graphical user interfaces for applications. | GTK2/2.24.33 |
| Ghostscript | Ghostscript is a versatile processor for PostScript data with the ability to render PostScript to different targets. | Ghostscript/9.56.1 |
| git | Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. | git/2.32.0
git/2.33.1 git/2.36.0 (D) |
| HarfBuzz | An OpenType text shaping engine | HarfBuzz/2.8.2
HarfBuzz/4.2.1 (D) |
| Hypre | Hypre is a library for solving large, sparse linear systems of equations on massively parallel computers. | Hypre/2.21.0-foss-2021a |
| ICU | ICU is a mature, widely used set of C/C++ and Java libraries providing Unicode and Globalization support for software applications. ICU is widely portable and gives applications the same results on all platforms and between C/C++ and Java software. | ICU/69.1
ICU/71.1 (D) |
| ImageMagick | ImageMagick® is a free, open-source software suite, used for editing and manipulating digital images. | ImageMagick/7.1.0-37 |
| Jasper | The JasPer Project is an open-source initiative to provide a free software-based reference implementation of the codec specified in the JPEG-2000 Part-1 standard | JasPer/2.0.33 |
| Java | Java Platform, Standard Edition (Java SE) lets you develop and deploy Java applications on desktops and servers. | Java/11.0.16 |
| KaHIP | The graph partitioning framework KaHIP -- Karlsruhe High Quality Partitioning. | KaHIP/3.14 |
| LAMMPS | LAMMPS is a classical molecular dynamics code with a focus on materials modeling. It's an acronym for Large-scale Atomic/Molecular Massively Parallel Simulator. | LAMMPS/23Jun2022 |
| LLVM | The LLVM Project is a collection of modular and reusable compiler and toolchain technologies. | LLVM/12.0.1
LLVM/14.0.3 (D) |
| LibTIFF | LibTIFF – Tag Image File Format(TIFF) Library and Utilities | LibTIFF/4.3.0 |
| LittleCMS | OPEN SOURCE small-footprint color management engine | LittleCMS/2.13.1 |
| Lua | Lua is a powerful, fast, lightweight, embeddable scripting language. Lua combines simple procedural syntax with powerful data description constructs based on associative arrays and extensible semantics. Lua is dynamically typed, runs by interpreting bytecode for a register-based virtual machine, and has automatic memory management with incremental garbage collection, making it ideal for configuration, scripting, and rapid prototyping. | Lua/5.4.4 |
| M4 | GNU M4 is an implementation of the traditional Unix macro processor. It is mostly SVR4 compatible although it has some extensions (for example, handling more than 9 positional parameters to macros). GNU M4 also has built-in functions for including files, running shell commands, doing arithmetic, etc. | M4/1.4.18
M4/1.4.19 (D) |
| METIS | METIS is a set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill reducing orderings for sparse matrices. The algorithms implemented in METIS are based on the multilevel recursive-bisection, multilevel k-way, and multi-constraint partitioning schemes. | METIS/5.1.0 |
| MPFR | The MPFR library is a C library for multiple-precision floating-point computations with correct rounding. | MPFR/4.1.0 |
| MUMPS | A parallel sparse direct solver | MUMPS/5.4.0 |
| Mako | Mako is a template library written in Python. It provides a familiar, non-XML syntax which compiles into Python modules for maximum performance. | Mako/1.1.4
Mako/1.2.0 (D) |
| Mesa | Mesa is an open-source implementation of the OpenGL specification - a system for rendering interactive 3D graphics. | Mesa/21.1.7
Mesa/22.0.3 (D) |
| Meson | Meson is an open source build system meant to be both extremely fast, and, even more importantly, as user friendly as possible. | Meson/0.58.2
Meson/0.62.1 (D) |
| NASM | NASM: General-purpose x86 assembler | NASM/2.15.05 |
| NCCL | The NVIDIA Collective Communications Library (NCCL) implements multi-GPU and multi-node collective | NCCL/2.10.3 |
| NLopt | NLopt is a free/open-source library for nonlinear optimization, providing a common interface for a number of different free optimization routines available online as well as original implementations of various other algorithms. | NLopt/2.7.1 |
| NSPR | ||
| NSS | ||
| NVHPC/22.7 |
Anaconda
Description: "Anaconda" (shortly "conda"), a Python package management, permits the creation of "environments," which are sets of modifiable packages. It accomplishes this by placing them in your residence. This page will take you through conda loading, environment creation, and modification so you may install and utilize any Python packages you require.
Usage: Anaconda could be loaded by: module load Anaconda3/2022.05
Creating the Conda Environment: Every user can create their own environments, and packages that are shared with the system-wide environments will not be reinstalled or copied to your file store; instead, they will be symlinked. This reduces the amount of space required in your /home directory to install numerous Python environments. To build a pristine environment with only Python 3.9 and numpy, execute: conda create -n mynumpy1 python=3.9 numpy To alter an existing environment, such as one of the anaconda installations, you can clone it: conda create --clone mynumpy2 -n mynumpy3
Package Installation Within a Conda Environment: After creating your own environment, you can install more packages or other versions of existing programs. There are two ways to accomplish this: conda and pip. If a package is accessible through conda, it is strongly advised that you use it to install packages. Using conda, you can search for packages: conda search pandas then download the package by following these steps: conda install pandas When attempting to install packages outside of your environment, you will receive a permission denied message. If this occurs, create or activate an environment you own. If a package is not available via conda, you can search for and install it using pip: pip search colormath pip install colormath
Usage of conda Environments: Once the conda module has been loaded, the required conda environments must be loaded or created. See the conda manual for documentation on conda environments. You can load a conda environment with the following: source activate mycondaenv where mycondaenv is the environment's name; unload one with: source deactivate which returns you back to the base environment. You can list all the accessible environments by using: conda env list A set of anaconda environments is provided system-wide; these are installed with the anaconda version number in the environment name and are never updated. Therefore, they will provide a fixed base for alternative environments or for direct use.
Using Conda in conjunction with the SLURM scheduler: Using the Anaconda batch mode To submit jobs to the Slurm job scheduler, you must run your primary application in batch mode from within your Conda environment. There are several stages to take: Create a script for application Create a Slurm script job to execute the application script sbatch is used to submit the job script to the task scheduler. Your application script should provide the necessary sequence of commands for your analysis. A Slurm job script is a Bash shell script of a particular type that the Slurm job scheduler recognizes as a job. Create a batch job submission script like the following and name it myscript.slurm: <clippy show="true">
- !/bin/bash
- SBATCH --partition=NVIDIA
- SBATCH --nodes=1
- SBATCH --ntasks=1
- SBATCH --cpu-per-task=8
- SBATCH --mem=16GB
- SBATCH --time=1:00:00
module purge eval $(conda shell.bash hook) conda activate myenvironment python script.py </clippy> The following describes each line: Command or Slurm option Description
- !/bin/bash
Use BASH to execute the script
- SBATCH
Syntax that allows SLURM to read your requests (ignored by BASH) --partition=NVIDIA Submit job to the NVIDIA partition --nodes=1 Use only one compute node --ntasks=1 Run only one task --cpus-per-task=8 Reserve 8 CPUs for the user --mem=16GB Reserve 16GB of RAM --module purge Purge or clear environment modules --time=1:00:00 Reserve resources for an hour eval $(conda shell.bash hook) Initialize the shell to use Conda conda activate myenvironment Activate your Conda environment, i.e., myenvironment python script.py Use python to run script.py Be sure to alter the requested resources based on your needs, but keep in mind that requesting fewer resources will reduce the waiting time for your work. To fully exploit the resources, particularly the amount of cores, it may be necessary to modify your application script. You can build application scripts and job scripts on your local machine and then move them to the cluster, or you can develop them directly on the cluster using one of the various text editor modules (e.g., nano, micro, vim). Submit the job to the job scheduler using the Slurm's sbatch command: sbatch myscript.slurm To determine the status of your position, enter: myqueue If there is no job status listed, then the job has been finished. The job's findings will be logged and, by default, stored in a plain-text file of the following format: slurm-<jobid>.outin in the identical directory from which the job script was submitted. To access this file's contents, enter: less slurm-<jobid>.out then press q to close the viewer.
CUDA[edit | edit source]
Description: Nvidia created the parallel computing platform and programming model known as CUDA for use with its GPUs for general computing (graphics processing units). By utilizing the capability of GPUs for the parallelizable portion of the calculation, CUDA enables developers to accelerate computationally heavy applications.
Usage: module load CUDA/11.4.1 To check if CUDA has been loaded, type: nvcc --version
GCC
The GNU Compiler Collection, commonly known as GCC, is a set of compilers and development tools available for Linux, Windows, various BSDs, and a wide assortment of other operating systems. It includes support primarily for C and C++ and includes Objective-C, Ada, Go, Fortran, and D. The Free Software Foundation (FSF) wrote GCC and released it as completely free (as in libre) software.
GCC is a toolchain that compiles code, links it with any library dependencies, converts that code to assembly, and then prepares executable files. It follows the standard UNIX design philosophy of using simple tools that perform individual tasks well. The GCC development suite utilizes these discrete tools to compile software.
When you run GCC on a source code file, it first uses a preprocessor to include header files and discard comments. Next, it tokenizes the code, expands macros, detects any compile-time issues, then prepares it for compilation. It is then sent to the compiler, which creates syntax trees of the program’s objects and control flow and uses those to generate assembly code. The assembler then converts this code into the binary executable format of the system. Finally, the linker includes references to any external libraries as needed. The finished product is then executable on the target system.
GCC examples
Compiling a program with GCC can be a straightforward matter/
gcc hello.c -o hello
Running this command processes the hello.c file and generates a binary called “hello”. Additional parameters can be passed.
gcc hello.c -O3 -o hello
In this example, the optimization parameter is set to 3, leading to more optimized code generation. Additional libraries can be included as well.
gcc hello.c -lncurses -o hello
This example includes the ncurses library.
More complex compilations are managed by Makefiles and are invoked with the “make” command.
Additional information
GNU Compiler Collection official page: https://gcc.gnu.org/
GCC Online documentation: https://gcc.gnu.org/onlinedocs/