oneAPI Collective Communications Library (oneCCL)

Installation   |   Usage   |   Release Notes   |   Documentation   |   How to Contribute   |   License

oneAPI Collective Communications Library (oneCCL) provides an efficient implementation of communication patterns used in deep learning.

oneCCL is governed by the UXL Foundation and is an implementation of the oneAPI specification.

Table of Contents

• oneAPI Collective Communications Library (oneCCL)
  • Installation
    • Linux build:
  • Usage
    • Launching Example Application
    • Common problems
  • Governance

SYCL support

oneCCL supports SYCL. In order to enable it, compile with icpx/icx-cl compiler or any compiler supporting find_package(IntelSYCL) in cmake.

To install Level Zero, refer to the instructions in Intel(R) Graphics Compute Runtime repository or to the installation guide for oneAPI users.

Installation

Linux build:

git submodule init # The two steps are not required if -DONECCL_USE_SYSTEM_LIBCCL=ON
git submodule update

mkdir build
cd build

cmake .. -DCMAKE_CXX_COMPILER=icpx -DCMAKE_C_COMPILER=icx -DCMAKE_BUILD_TYPE=release -DCMAKE_INSTALL_PREFIX=$(pwd)/_install
make -j install

DONECCL_USE_SYSTEM_LIBCCL option can be used to use libccl implementation from the system rather than compiling it from sources ar deps/libccl To compile legacy libccl code and use it with the flag use following steps:

git clone https://github.com/intel-innersource/libraries.performance.communication.oneccl ccl

git clone https://github.com/intel-innersource/libraries.performance.communication.oneccl-v2 ccl-v2

# Build legacy libccl with SYCL support
cd ccl
mkdir build
cd build
cmake .. -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DCMAKE_EXPORT_COMPILE_COMMANDS=1 && make -j install

# Build legacy libccl for CPU only
cd ..
mkdir build_cpu
cd build_cpu
cmake .. && make -j install
cd ..

# Copy CPU only binaries of legacy libccl
mkdir -p build/_install/lib/ccl/cpu/
cp -r build_cpu/_install/lib build/_install/lib/ccl/cpu/

# Go back to the new repo, assuming both repos were cloned at `~`
cd ~/ccl-v2
mkdir build
cd build
# Source previously build libccl, so CMAKE_PREFIX_PATH is set to `~/ccl/build/_install/lib/cmake`
source ~/ccl/build/_install/env/vars.sh
# Run cmake, please note DONECCL_USE_SYSTEM_LIBCCL=ON to use oneCCL through cmake's `find_package`
cmake .. -DCMAKE_CXX_COMPILER=icpx -DCMAKE_C_COMPILER=icx -DONECCL_USE_SYSTEM_LIBCCL=ON -DCMAKE_INSTALL_PREFIX=$(pwd)/_install
make install -j

Additional flag for ARC and BMG GPU

if you build this package for ARC or BMG GPU with embedded libccl(DONECCL_USE_SYSTEM_LIBCCL=OFF), you need to add an additional flag during the cmake process. Commands below show the build process.

ARC GPU

git submodule init # The two steps are not required if -DONECCL_USE_SYSTEM_LIBCCL=ON
git submodule update

mkdir build
cd build

cmake .. -DCMAKE_CXX_COMPILER=icpx -DCMAKE_C_COMPILER=icx -DCMAKE_BUILD_TYPE=release -DCMAKE_INSTALL_PREFIX=$(pwd)/_install -DCCL_ENABLE_ARCA=ON
make -j install

BMG GPU

git submodule init # The two steps are not required if -DONECCL_USE_SYSTEM_LIBCCL=ON
git submodule update

mkdir build
cd build

cmake .. -DCMAKE_CXX_COMPILER=icpx -DCMAKE_C_COMPILER=icx -DCMAKE_BUILD_TYPE=release -DCMAKE_INSTALL_PREFIX=$(pwd)/_install -DCCL_ENABLE_ARCB=ON
make -j install

Usage

Launching Example Application

Use the command:

$ source <install_dir>/env/setvars.sh
$ CCL_LOG_LEVEL=info mpirun -n 2 <install_dir>/examples/sycl/sycl

Common problems

C API version of oneCCL introduced a new plugin based architecture. Running examples or tests might require adjusting LD_LIBRARY_PATH, so it points to at least one directory with correct plugin. It can be done using vars.sh script or manually like export LD_LIBRARY_PATH=<build_dir>/plugins/legacy/:$LD_LIBRARY_PATH, so test binaries linking to libccl.so.2 through RUNPATH can still discover required plugin.

You can also use CCL_PLUGIN environment variable to set name of the plugin (ONECCL_LEGACY/ONECCL_LEGACY_CPU/ONECCL_NULL) or a whole path to a shared object implementing oneCCL plugin interface (<build_dir>/plugins/legacy/libccl_legacy.so).

Governance

The oneCCL project is governed by the UXL Foundation and you can get involved in this project in multiple ways. It is possible to join the Special Interest Groups (SIG) meetings where the group discuss and demonstrates work using the foundation projects. Members can also join the Open Source and Specification Working Group meetings.

You can also join the mailing lists for the UXL Foundation to be informed of when meetings are happening and receive the latest information and discussions.

Contribute

See CONTRIBUTING for more information.

License

Distributed under the Apache License 2.0 license. See LICENSE for more information.

Security Policy

See SECURITY for more information.