
                        Score-P 1.4.2 OPEN ISSUES
                        =========================
                                                           Effective: Jun 2015

This file lists known limitations and unimplemented features of
various Score-P components.

--------------------------------------------------------------------------------

* Platform support

  - Score-P has been tested on the following platforms:
    + IBM Blue Gene/Q, only static libraries supported
    + IBM SP & BladeCenter clusters, only static libraries supported
    + Cray XC, only static libraries supported
    + SGI Altix/ICE
    + K Computer, Fujitsu FX10 and FX100 systems
    + various Linux/Intel (x86/x64/Power8/ARM AArch32/AArch64) clusters
    + Intel Xeon Phi, only native mode supported
    The provided configure options (see INSTALL) may provide a good
    basis for building and testing the toolset on other systems.

  - The following platforms have not been tested recently:
    + IBM Blue Gene/P
    + Cray XT, XE, XK
    + Sun Solaris/SPARC-based clusters
    However the supplied build system might still work on these
    systems.

  - The following platforms have not been tested:
    + NEC SX-9
    + IBM Blue Gene/L
    + SiCortex systems
    + other NEC SX systems

  - Each toolset installation can only support one MPI implementation
    (because MPI is only source-code but not binary compatible). If
    your systems support more than one MPI implementation (e.g. Linux
    clusters often do), separate builds for each MPI implementation
    have to be installed. The same applies for SHMEM.

  - The same is true if your system features more than one compiler
    supporting automatic function instrumentation.

  - To build Score-P it is required to have Fortran compilers and
    highly recommended to have MPI compilers.

--------------------------------------------------------------------------------

* Automatic compiler instrumentation via "scorep" based on (often
  undocumented) compiler switches

  - GNU    : tested with GCC 4 and higher
  - PGI    : tested with version 10.1 and higher
             Note that PGI 13.8 is currently not supported as it is not
             recognized as an PGI compiler anymore
  - Oracle : only works for Fortran (not C/C++), tested with version 12.2
  - IBM    : only works for xlc/xlC version 7.0 and xlf version 9.1 and
             higher and corresponding bgxl compilers on BlueGene systems
  - Intel  : only works with Intel icc/ifort version 11 and higher compilers
  - Cray   : tested with version 8.1.8 and later

  - The Intel compiler provides the function name and the source code
    location in one string separated by a colon.  Thus, if the path
    name of the source code file contains colons, Score-P will split
    the source file name and the function name improperly.
    Additionally, the provided function name makes it impossible to
    distinguish overloaded functions in C++. Thus, functions which
    differ only in the argument list will be mapped to the same
    function definition by Score-P.

  - When using any of the intrinsics headers from GCC (e.g.,
    xmmintrin.h and friends), than, from GCC version 4.6 onwards, it
    is known that the result of the compiler instrumentation will fail
    at link time because of undefined reference. A work around is to
    pass

      -finstrument-functions-exclude-file-list=intrin.h

    as flag to GCC on the command line, or pass it as configure option
    to Score-P with

      --with-extra-instrumentation-flags=-finstrument-functions-exclude-file-list=intrin.h

    to make it permanently inside the Score-P installation.

  - Because the compiler instrumentation interface of the XL compiler
    series changed with xlc 11.1/xlf 13.1. You need a separate
    installations for older versions than xlc 11.1/xlf 13.1 and newer
    versions.

  - Measurement filtering can only be applied to functions
    instrumented by the IBM, GNU, Intel, PGI, or Oracle compilers as
    well as functions instrumented by PDT and user functions.
    Filtering of MPI, SHMEM, and OpenMP runtime functions is always
    ineffective.

  - The GNU Fortran compiler versions 4.6.0 and 4.6.1 have a bug which
    leads to an internal compiler error when using automatic function
    instrumentation.  It is therefore recommended to either use an
    older/newer version of the compiler or to work around this issue
    by using manual instrumentation or automatic source-code
    instrumentation based on PDToolkit.

  - On systems where libiberty is build without -fPIC, one cannot use
    libbfd for compiler instrumentation's region naming (GNU, Intel,
    Cray) in --enable-shared builds of Score-P. Either provide a
    libbfd that comes with a proper libiberty (--with-libbfd) or
    disable libbfd region naming (--without-libbfd). In the latter
    case, region names will be provided by nm.

  - The GNU compiler instrumentation provides only functions pointers,
    which we look up in the symbol table of the executable. Thus,
    functions from shared libraries, will not appear (are
    automatically filtered) in the measurement output.

  - Due to constant changes in the plug-in API of the GNU compiler
    infrastructure, it is highly unlikely that the Score-P instrumentation
    plug-in builds with version newer than GCC 5.

  - The pgCC compiler versions 13.9 and higher preinclude omp.h for
    OpenMP codes. This results in multiple defined symbols if the
    source file is preprocessed before compilation. Since version 14.1
    an option is available to avoid preinclusion, which we can use for
    preprocessed source files.  For the pgCC versions 13.9 until 14.0,
    preprocessing is not possible for C++ codes.

  - The GCC compiler uses DWARF v4 as the default debug information
    format since version 4.8. But when used with an older binutils
    version this results in unreliably region meta data. If file name
    information are missing in the resulting region meta data, than
    try recompiling with the -gdwarf-3 or -gdwarf-2 flag.

  - When instrumenting an application which contains bison generated
    code, the link step fails because of multiple defined symbols
    conflicting with the online access module of Score-P. Please use
    'scorep --noonline-access' to disable the online access and avoid
    multiple symbol definitions.

--------------------------------------------------------------------------------

* MPI support

  - Functions added with MPI-3.0 are not supported by Score-P.
    Measurements using libraries providing MPI-3.0 functions will be
    incomplete if any of the new functions are used, and use of
    communicators or windows created by new functions are likely to
    result in measurement failure.

  - C++ bindings for MPI are not supported. These were deprecated as
    of MPI-2.2 and removed with MPI-3.0. When using C++ bindings for
    MPI, Score-P will most likely indicate the failed library wrapping
    with the following warning: "If you are using MPICH1, please
    ignore this warning. If not, it seems that your interprocess
    communication library (e.g., MPI) hasn't been initialized. Score-P
    can't generate output."
    If your MPI implements the C++ bindings in a separate library on
    top of the C bindings, the following workaround might work for
    you:
    1) Instrument your application with 'scorep --keep-files -v' It
       will make the instrumenter not remove the intermediate files
       and output the commands it executes.
    2) Copy the link command (the last command from the scorep
       output).
    3) Insert a link flag (e.g. -lmpicxx) for your C++ bindings
       library right before '-lscorep_adapter_mpi_events' and execute
       the modified link command.

  - When using derived data types in non-blocking communications, and
    no support for MPI_Type_dup() was detected, please ensure that the
    MPI_Datatype handle is still valid at the time the request
    finishes.

  - Online detection of MPI wait states might produce wrong results
    when messages sent within different communicators overtake each
    other.

  - Currently, Score-P can't handle MPI_Finalize() calls that occur
    after the end of main(), e.g., via a destructor of a static C++
    object. Please call MPI_Finalize() before the end of main(). The
    issue will be resolved in a future version of Score-P.

  - If an application uses MPI inter-communicators, Score-P
    measurement will hang during the creation of the communicator.

  - The IBM Platform MPI (not mpixlc!) compiler wrapper (the formerly
    HP-MPI) does not append its libraries at the very end of the
    original link command. Thus, instrumenting applications with
    Score-P fails at link time due to unresolved symbols in the
    Score-P libraries.

--------------------------------------------------------------------------------

* OpenMP support

  - Function instrumentation using the Intel compiler version 11.1 for
    codes using OpenMP tasking is erroneous. You might try PDT
    instrumentation instead.

  - When compiling with the PGI compiler version 10.1, local variables
    that are defined after a OpenMP for construct share the same memory
    address. This breaks the OPARI2 instrumentation for task tracking
    and may lead to segmentation faults in the measurement system.
    Our recommendation is to use a newer compiler version, According
    to our tests, late compiler versions have fixed this issues.
    We tested with PGI compiler version 11.7.

  - Currently, the instrumenter allows to switch off OPARI2
    instrumentation for OpenMP programs. If OpenMP parallel regions
    are not instrumented and an event occurs inside a parallel region
    (e.g., because an instrumented function is called inside of a
    parallel region) Score-P will crash. We are considering to make
    the usage more secure in future versions of Score-P. This might
    result in a change of the instrumenter option or the option may
    vanish completely. However, there are some use-cases where
    disabling OPARI2 instrumentation is helpful.

  - Due to a bug in the Cray compilers OpenMP instrumentation is
    broken if an OpenMP parallel pragma is used in combination with
    an if clause.

 - OPARI2-instrumented Fortran OpenMP codes that use compiler options
   to change the default name-mangling (XL compilers: -qextname, GNU:
   -fno-underscoring' and '-fsecond-underscore') will likely crash.
   This is because a variable name used in the Score-P libraries does
   not match the one instrumented by OPARI2. On AIX, a workaround is
   to use scorep's commandline option --thread=omp:ancestry or to
   manually rename it at link time (-brename:pomp_tpd_,pomp_tpd).

 - With PGI C++ v13.10 compiler, preprocessing of OpenMP codes using
   OPARI2 is not possible any longer as the compiler itself adds a
   '--preinclude omp.h' option to the call of pgcpp1. This leads to
   'invalid redeclaration' errors. As a workaround, use the
   '--nopreprocess' instrumenter option.

 - The OPARI2 instrumentation and the preprocessing for OPARI2
   instrumentation prepend some headers to source files which
   include stdint.h in C/C++ files. The behavior of this header changes
   whether macros the __STDC_CONSTANT_MACROS or __STDC_LIMIT_MACROS are
   defined. If these macros are defined in a header file or source file,
   the instrumentation will prepend the include directive before the macro
   definition. Thus, macros like UINT32_C are left undefined. As workaround,
   pass macros like __STDC_CONSTANT_MACROS or __STDC_LIMIT_MACROS on the
   command line. See also the Open Issues of OPARI2.

--------------------------------------------------------------------------------

* CUDA support

  - CUDA device activities get lost for CUDA 5.5 (CUPTI 4). The last
    activity in a CUPTI activity buffer gets lost, when the buffer is
    full. The issue can be avoided by providing buffers, which are
    large enough to store all CUDA device activities until the buffer
    is flushed. The user can specify the CUPTI activity buffer (chunk)
    size with the environment variable SCOREP_CUDA_BUFFER_CHUNK. In
    CUDA 6.0 (CUPTI 5) this issue is fixed.
  - Support for CUDA 4.2 and prior versions is deprecated. Score-P may
    work in combination with these CUDA versions but it is not tested.
    Support for CUDA 4.2 and prior versions will become officially
    unsupported in a future release of Score-P.

--------------------------------------------------------------------------------

* SHMEM support

  - When using the OpenSHMEM reference implementation and building
    Score-P as a shared library, ensure that the GASNet library is
    build with -fPIC on platforms which need this flag for shared
    library code. For example, run make with the MANUAL_LIBCFLAGS and
    MANUAL_CFLAGS variables set:

      make MANUAL_LIBCFLAGS="-fPIC -DPIC" MANUAL_CFLAGS="-fPIC -DPIC"

    on GNU/Linux platforms.
    Also, if you encounter segmentation faults when running the
    instrumented application with Score-P attached, but the error
    disappears magically on subsequent runs, than the OpenSHMEM
    reference implementation may not consider some of your global or
    static variables as symmetric objects.  Please allocate these
    objects with shmalloc() to ensure that they are in the symmetric
    heap.
  - Currently tested SHMEM implementations:
    + OpenSHMEM reference implementation 1.0f
    + Open MPI implementation 1.8.2a
    + SGI SHMEM
    + Cray SHMEM
  - The online access interface is currently not available in SHMEM
    applications.

--------------------------------------------------------------------------------

* POSIX threads support

  - Please note that you need to instrument every thread creation
    where the newly generated thread might create measurement events
    (which is usually the case when you use the default compiler-based
    instrumentation).

  - The usage of pthread_detach will cause the program to fail if the
    detached thread is still running after the end of main.

  - Pthread support is currently not available on systems that use
    linkers other than the GNU linker, e.g., AIX systems.

  - Please note that on systems where Pthreads don't need extra flags
    (like e.g., -pthread) to be compiled and linked (e.g., Cray, Blue
    Gene/Q), Score-P can't instrument Pthreads automatically. You need
    to enable Pthread instrumentation manually via scorep's
    --thread=pthread option.

--------------------------------------------------------------------------------

* Score-P misc.

  - Score-P does not support MPMD style programs where the executables
    are instrumented differently. I.e., it is best to compile/link all
    sources with the same set of flags for the Score-P instrumenter, so
    that the instrumenter itself can't automatically decides what
    paradigms the program uses. This applies to the --mpp and --thread
    flags.

  - There might be a performance impact when instrumenting code
    without explicitly given optimization flags. The instrumenter adds
    compiler flags to enable additional debugging
    information. Depending on the compiler this may turn off
    optimization unless optimization flags are explicitly specified.

  - Applications using an autotools-based buildsystem should add the
    option "--disable-dependency-tracking" to the usual configure
    call. Score-P is added only at make time like "make CC="scorep
    <your-CC>" (same for CXX, F77, and FC)

  - Literal file-filter rules like "INCLUDE bt.f" for Fortran files
    that will be processed by OPARI2 (i.e., files containing OpenMP or
    POMP user pragmas) do not work as expected as OPARI2 changes the
    file name (here to bt.opari.f)

  - Due to a bug in PDT 3.18 and earlier versions, PDT support is
    disabled on BlueGene systems.

  - Rusage-based metrics are not supported on BlueGene systems.

  - Using scorep as a compiler/linker preposition is sometimes not
    supported by application build systems, e.g., CMake.  In such
    cases, it may be appropriate to create scripts which invoke the
    compiler/linker with the scorep preposition defined.  For example,
    specify scorep_ftn as the compiler and linker during application
    configuration, where this is a script executing your Fortran
    compiler as "scorep ftn $*".

  - COBI binary instrumentation is currently disabled due to changes
    in Score-P's linking process that now passes more libraries to
    COBI/Dyninst than it can handle at the moment. This issue might be
    addressed in a future release of Score-P and/or Dyninst.

  - Traces generated by applications compiled with the CCE Fortran
    compiler on Cray X series systems are inconsistent as there is no
    exit event for 'main'. I.e., such traces are not analyzable by
    Scalasca.

  - Running make check fails if SCOREP_EXPERIMENT_DIRECTORY is set to
    'scorep'. One workaround is to run '(unset
    SCOREP_EXPERIMENT_DIRECTORY; make check;)'.

  - Currently OTF2 definitions are limited with respect to the maximum
    number of group members (i.e., MPI_COMM_WORLD). OTF2 traces with
    more than 512k threads are not supported yet.

  - In some cases the PGI 11.7 compiler fails to link C++ programs
    that use user-instrumentation macros due to an undefined reference
    to '_Unwind_Resume'. Adding -lstdc++ to the original link line
    solves this issue.

  - In cases where scorep was compiled with PAPI support, the
    instrumenter option --static (only available for '--enable-shared
    --enable-static' builds) explicitly adds the static PAPI library to
    the link line which may cause 'undefined reference' errors. This
    may happen for other libraries that were requested at configure time
    as well.

--------------------------------------------------------------------------------

Please report bugs, wishes, and suggestions to <support@score-p.org>.
