OS X 10.11 (El Capitan) is not compatible with any version of the LSST
software stack, up to and including Summer 2015. This is due to the
new System Integrity Protection (SIP)
feature. As well as preventing anyone from modifying system files,
protected binaries no longer inherit linker environment variables. In
DYLD_LIBRARY_PATH is ignored. The stack and EUPS
completely rely on this environment variable and without it any
packages using C++ will not import.
A normal stack build of Summer 2015 fails almost immediately in the
base package as that is the first that attempts to import C++
library code into python.
SIP only affects Apple-supplied binaries. For the stack the issue is
that Python scripts and scons are always run with a shebang (
#! /usr/bin/env python. Since env is in
/usr/bin it is
covered by SIP protections such that the library load path environment
variable is stripped before being executed. scons is executed via
env and runs the tests in a subprocess which will inherit a
stripped environment and will therefore fail. Furthermore, executable
scripts in the
bin directory will also have the environment
stripped if those scripts are executed via env, and will therefore
fail to load C++ python modules.
The first release with a functioning stack on El Capitan was
Changes to the Stack¶
The following changes were required to enable the LSST software to build on El Capitan:
- Modify EUPS table files to introduce a new environment variable,
LSST_LIBRARY_PATH, that looks identical to
DYLD_LIBRARY_PATHbut which is not intercepted by SIP. This variable is used inside scons to ensure that all tests are executed with the correct environment enabled (scons launches tests as sub processes).
- /usr/bin/env can no longer be used to run scripts from the
command line. The shebang line must point to an explicit executable
and that executable can not be in
/bin. For Python scripts the shebang must point to a user-installed Python binary. To allow the rewriting of the shebang to occur a new scons build target has been created,
shebang, that will copy files from a
bin.srcdirectory to a
bindirectory, modifying them during the copy. The rewriting does not happen on all platforms (although that is not guaranteed behavior for the future) and only files requiring rewrites should be placed in that directory.
The reason for the new environment variable specifically for running tests is that it is difficult to ensure that the build is being triggered with every parent process being correctly configured to pass through the library path. At the very least we would have to fix eups, scons and lsstsw and even so any shell scripts that people may use to trigger builds will also have their environment stripped.
One additional complication on El Capitan is that Apple no longer
distributes the OpenSSL include files. Apple deprecated the use of
OpenSSL in OS X 10.7 (Lion) and removed the include files in El Capitan
(the libraries remain for binary compatibility). The
libevent packages were modified to
disable the use of SSL on OS X. 
At the time of writing
lsst_distrib builds correctly on El Capitan.
One other approach was considered and that was to copy /usr/bin/env to a new location and change every script to use the new env. This would have worked because the copied env would no longer be susceptible to SIP restrictions. The consensus was that this solution of a new env did not feel acceptable and would require too many edge cases in the documentation.
Porting to El Capitan¶
For developers the following must be remembered when modifying packages:
- Ensure that
DYLD_LIBRARY_PATHappears in a table file.
- Python scripts should be placed in the
bin.srcdirectory and not the
bindirectory. A suitable
SConscriptfile is shown at the end of this document and can also be found in the package template repository.
- People can no longer build or use the stack with the system Python.
- Executable shell scripts should ensure they run setup rather than
relying on the setup of the parent shell. This is because
DYLD_LIBRARY_PATHwill no longer be guaranteed to be set in the subshell. For an explicit discussion of this see Example SIP Behavior.
- If a package requires OpenSSL, consider supporting both OpenSSL and Apple CommonCrypto. Otherwise OpenSSL may have to be made an explicit prerequisite on OS X.
The changes to allow tests to correctly inherit the environment only
affect packages built using
sconsUtils. Two packages are known not
to work on El Capitan:
sconsUtilsin a non-standard way such that most of the targets are hand-crafted. The test target does not use the
sconsUtilstest framework so all the tests fail.
qservuses a bespoke scons configuration system that may need to be taught how to inherit
LSST_LIBRARY_PATHfor the test environment. Additionally
qservuses OpenSSL when calculating digests and these will have to be ported to CommonCrypto.
Relevant JIRA Tickets¶
Example SIP Behavior¶
The following code
#! /usr/bin/env python import os print(os.environ["DYLD_LIBRARY_PATH"])
KeyError on El Capitan. Running it as
python test.py correctly prints the value of the environment variable.
Similarly shell scripts, which always tend to use shells from
/usr/bin, will therefore also lose
DYLD_LIBRARY_PATH. This script:
#!/bin/bash echo DYLD: $DYLD_LIBRARY_PATH echo LSST: $LSST_LIBRARY_PATH
will only result in values appearing from the second line. One solution is to explicitly set the path at the start of the script:
#!/bin/bash # On OS X El Capitan we need to pass through the library load path if [[ $(uname -s) = Darwin* ]]; then if [[ -z "$DYLD_LIBRARY_PATH" ]]; then export DYLD_LIBRARY_PATH=$LSST_LIBRARY_PATH fi fi
The following code can be used in the
bin.src directory to configure scons:
from lsst.sconsUtils import scripts scripts.BasicSConscript.shebang()
|||The LSST stack does not use SSL capabilities in
|||Interestingly, if the shebang is removed and replaced with a blank line, the environment is inherited without being filtered by the default POSIX shell.|
OS X equivalent of
LD_LIBRARY_PATH. Specifies the search path for loading shared libraries.