Build Notes: S/PHI/nX on Windows

Written by sixten on . Posted in All Posts, S/PHI/nX

Microsoft Windows

S/PHI/nX and its underlying high-performance general-purpose C++ class library SxAccelerate are already available on all popular Linux derivates (such as RHEL, CentOS, Fedora, OpenSuSE, Debian, Ubuntu) as well as MacOS X and FreeBSD.

However, until now the support of Windows was missing. In particular, having the general-purpose C++ library available under all relevant operating systems would be very beneficial:

SxAccelerate provides an intuitive high-level C++ programming interface necessary for rapid application development. It is based on modern programming techniques to allow the generation of very efficient code. Furthermore, it interfaces most of the commonly needed operating system function calls:

  • Multithreading:
    S/PHI/nX supports an intuitive interface for programming with multiple threads. A built-in thread manager spools the necessary tasks to the underlying thread mechanism of the operating system.
  • File system operations:
    While every operating system has different ways of handling file system operations SxAccelerate provide a specify well-defined way. For example, relative symbolic links are defined differently among different Linux derivates. By developing file system operations in SxAccelerate the resulting links always behaving identical on any OS.
  • Data containers:
    SxAccelerate comes with a set of data containers such as SxArray, SxList, SxUniqueList, or SxMap. They are very intuitively to use but the underlying implementation is efficient with respect to both memory consumption and performance.
  • Fully automatic memory management:
    All data types in S/PHI/nX such as SxString, SxArray, SxVector, SxMatrix, etc. automatically deallocate memory as soon as it is possible. This handling is efficient – no (slow) garbage collecting is applied.
  • Algebraic Expressions:
    SxAccelerate provides a language to express algebraic expressions efficiently on modern computer platforms.
    For example, the algebraic expression
    \(\mathbf{M} = \alpha \mathbf{A}^T\mathbf{B} + \beta \mathbf{C}\)
    can be implemented as

    M = a*A.transpose() ^ B + b*C;

    At compile time the above expression is automatically mapped to high-performance BLAS/LAPACK function calls.

Until now, all of the above features were available on all Linux derivates as well as Mac OSX. Porting the above features to Windows in the same level of efficiency requires focussing on various details:

  • S/PHI/nX and SxAccelerate can be compiled with any compiler (GNU, Intel, etc.). In order to provide a native Windows executable the Microsoft Visual Studio compiler should be applied. 
  • The make environment of S/PHI/nX and SxAccelerate is based on the autotools tool chain. The Microsoft Visual Studio compiler uses Projects instead.
  • On Linux / MacOS X the pthreads library is used to support multithreading. While pthreads is also available on Windows, its 64bit implementation still seem to have issues on Windows. Therefore, the native Windows threads should be supported by SxAccelerate
  • The Windows file system is fundamentally different to that of Linux / MacOS X

Recently Gemmantics ported successfully S/PHI/nX and SxAccelerate to Windows. The necessary code modifications have been submitted to the S/PHI/nX repository (branch: gemmantics/devel) and will soon be merged to the main trunk.

This blog entry will explain in detail how to prepare a Windows machine (bare metal or Virtual Machine) to build S/PHI/nX on Windows.

Installing necessary Software Packages

The following discussion assumes a computer or virtual machine being installed with Windows 7 Professional (64bit). Other Windows versions might work just as well but we didn’t test them yet. Make sure that the product key of Windows has been activated.

In the following steps administrator privileges are required to install the necessary software packages:

  1. Microsoft Visual Studio (Express) 2012 for Desktop (MSVC)
  2. MinGW / MSYS (flex, bison)
  3. Intel Math Kernel Library (Intel MKL)
  4. optional: Subversion

Microsoft Visual Studio Express 2012 for Desktop (MSVC)

The Visual Studio Express is the free-of-charge version of Visual Studio. It can be obtained from http://www.microsoft.com/express. Microsoft requires a registration of the software during the first launch of Visual Studio Express. Of course, the the fully-featured commercial version Visual Studio 2012 can also be used to build S/PHI/nX.

MSVC

Visual Studio (Express)  2012 provides a tool called lib.exe which will be needed later on. Please add it to the binary search path %PATH% as follows:

  1. START > Control Panel > System > Advanced Settings > Environment Variables
  2. Edit “System Variables” > “Path”
  3. Append:
    ;/c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 11.0/VC/bin;/c/Program\ Files\ \(x86\)/Microsoft\ Visual\ Studio\ 11.0/Common7/IDE

Subversion

Download the subversion client from http://sourceforge.net/projects/win32svn/files/latest/download.

MinGW/MSYS

While MinGW / mSYS actually provide a GNU C/C++ compiler S/PHI/nX will be compiled with the Microsoft Visual Studio (Express) compiler. However, MinGW/mSYS provides a very useful environment for people used to work with Linux / MacOS X. Furthermore, to build SxAccelerate the flex and bison have to be available has they will be incorporated into the Visual Studio build chain. Please execute the following steps:

  1. Download the MinGW installer from http://sourceforge.net/projects/mingw
  2. Execute
    mingw-get-inst

    (typically located in C:\administrator\Downloads)

  3. Select: (i) C, (ii), C++, and (iii) MSYS Base
  4. Start a MinGW Shell: START > All Programs > MinGW > MinGW Shell
  5. Install the following MinGW components:
    mingw-get install msys-flex
    mingw-get install msys-bison
    mingw-get install msys-libtool
    mingw-get install msys-patch
    mingw-get install g++
    mingw-get install msys-wget
    mingw-get install msys-vim
  6. Add MinGW’s path to the binary search path of all users:
    START > Control Panel > System > Advanced Settings > Environment Variables
    Edit “System variables” > Path * Append:

    ;c:\mingw\msys\1.0\bin

Preparing the Environment

In order to provide a multi-user S/PHI/nX build environment it is recommended to prepare a shared folder on a file server, such as H:\common. Therefore, mount the folder H:\common or create a local folder, e.g., in MinGW shell:

mkdir p /h/common

Please register the new path for all users by setting the environment variable SXNUMLIBS:

  • START > Control Panel > System > Advanced Settings > Environment Variables
  • System Variables > New
  • SXNUMLIBS = h:\common

If H:\common is not an appropriate location on your system specify the proper path in SXNUMLIBS and replace the path in the following steps accordingly.

3rd-party Libraries

In an MinGW shell execute the following commands

bash> mkdir devel; cd devel
bash> svn co --username johndoe https://SXLIB.REPO.URL/repo/url/sphinx
bash> cd sphinx/sxaccelerate
bash> ./setup
bash> cd 3rd-party
bash> ./configure --disable-atlas --disable-lapack --disable-fftw --enable-regex --enable-pthreads --enable-dll --enable-shared --prefix=/h/common
bash> make all install

Please convert the obtained netCDF libraries to MSVC import libraries as follows:

bash> mkdir devel; cd devel
bash> cd /h/common/lib
bash> lib /machine:i386 /def:netcdfdll.def /out:netcdf.lib
bash> cd /h/common/bin
bash> mv libnetcdf-7.dll netcdfdll.dll

Intel MKL

Gemmantics is Intel Software Partner

If SxAccelerate should be built with numeric support a BLAS/LAPACK and FFT library has to be provided. As Intel Software Partner Gemmantics recommends the application of the Intel Math Kernel Library (Intel MKL). The MKL is an extremely efficient and highly portable numeric library with BLAS/LAPACK as well as FFT interfaces.

Intel MKL is a commercial library which has to be purchased by Intel. After obtaining the Intel MKL package for Windows simply execute the installer w_mkl_AA.B.CC.DDD.exe with AA.B.CC being the MKL version and DDD denoting the patch level.

Move the following libraries to h:\common

bash> mkdir devel; cd devel
bash> mkdir /h/common/mkl
bash> cp -r /c/Program\ Files\ \(x86\)/Intel/Composer\ XE/mkl/include/* /h/common/include
bash> cp -r /c/Program\ Files\ \(x86\)/Intel/Composer\ XE/mkl/lib/ia32/* /h/common/lib

Building SxAccelerate

With the above environment configured building SxAccelerate is straightforward.  Simply open the file browser (WIndows Explorer) and navigate to the SxAccelerate source tree sphinx/sxaccelerate/src/windows. This folder contains a ready-to-use MSVC project file.

SxAccelerate

The project file is generated under Linux or Mac OSX by the S/PHI/nX make environment. When modifying the file structure of S/PHI/nX or SxAccelerate simply run make filelist to update the automake environment and make msfilelist to update the MSVC project.

Open the project file by double-clicking the file SxAccelerate. Since the code has not been digitally signed a warning message will appear and asks whether to import the project. After confirmation the SxAccelerate project folder will open.

In MSVC the compilation can be triggered by hitting F7.

The compiled libraries and executables can be found in sphinx/sxaccelerate/bin/Debug.

Building S/PHI/nX

Once SxAccelerate has been successfully compiled it is time to build S/PHI/nX.  Similar to SxAccelerate there is also a MSVC project file available for S/PHI/nX. It is located in sphinx/src/windows. Open it by double-clicking the icon. In MSVC the compilation can be triggered by hitting F7.

The compiled libraries and executables can be found in sphinx/bin/Debug.

Below please find some screenshots of the Windows session.

Conclusion

With the new Windows port provided by Gemmantics the general purpose C++ class library SxAccelerate became a truly cross-platform library which supports

  • S/PHI/nXLinux (RHEL, CentOS, Fedora, Debian, OpenSuSE)
  • MacOS X
  • FreeBSD
  • AIX, IRIX, HP-UX,
  • and now Windows.

Applications developed based on the SxAccelerate libraries will be highly efficient due to its underlying BLAS/LAPACK/FFT function call mapping, very stable due to its built-in consistency checking mechanisms, scaling with its new multithreading layer, and available on all major platforms.

Please follow the author on Twitter @SxBoeck.

© 2013 Gemmantics