BUILD: macOS build broken #115

Affected version: v3.2.0-RC15
Env: macOS

Reproduction

Building the stack as per: https://dev.pep.foundation/Adapter/Adapter%20Build%20Instructions_Version_3.x_DRAFT

Engine build: only changed 'DEBUG=' to 'DEBUG=debug' in local.conf.
Problem does not appear with 'DEBUG='.
Reliably reproducable.

Observed Behaviour

Upon loading of the pEpEngine.dylib by any program linking dynamically to it:

dyld[31920]: Library not loaded: 'libpEpEngine.dylib'
  Referenced from: '/Users/heck/src/pEp/libpEpAdapter32/test/test_adapter_cxx'
  Reason: tried: '/Users/heck/local-3x/lib/libpEpEngine.dylib' (code signature does not cover entire file up to signature), '/libpEpEngine.dylib' (no such file), 'libpEpEngine.dylib' (no such file), '/usr/local/lib/libpEpEngine.dylib' (no such file), '/usr/lib/libpEpEngine.dylib' (no such file), '/Users/heck/local-3x/lib/libpEpEngine.dylib' (code signature does not cover entire file up to signature), '/libpEpEngine.dylib' (no such file), '/Users/heck/src/pEp/libpEpAdapter32/test/libpEpEngine.dylib' (no such file), '/usr/local/lib/libpEpEngine.dylib' (no such file), '/usr/lib/libpEpEngine.dylib' (no such file)
Abort trap: 6

Expected Behaviour

Build succeeds and dylib loads as usual

Impact

High - i need debugging symbols

Affected version: v3.2.0-RC15 Env: macOS ### Reproduction Building the stack as per: https://dev.pep.foundation/Adapter/Adapter%20Build%20Instructions_Version_3.x_DRAFT Engine build: only changed 'DEBUG=' to 'DEBUG=debug' in local.conf. Problem does not appear with 'DEBUG='. Reliably reproducable. ### Observed Behaviour Upon loading of the pEpEngine.dylib by any program linking dynamically to it: ``` dyld[31920]: Library not loaded: 'libpEpEngine.dylib' Referenced from: '/Users/heck/src/pEp/libpEpAdapter32/test/test_adapter_cxx' Reason: tried: '/Users/heck/local-3x/lib/libpEpEngine.dylib' (code signature does not cover entire file up to signature), '/libpEpEngine.dylib' (no such file), 'libpEpEngine.dylib' (no such file), '/usr/local/lib/libpEpEngine.dylib' (no such file), '/usr/lib/libpEpEngine.dylib' (no such file), '/Users/heck/local-3x/lib/libpEpEngine.dylib' (code signature does not cover entire file up to signature), '/libpEpEngine.dylib' (no such file), '/Users/heck/src/pEp/libpEpAdapter32/test/libpEpEngine.dylib' (no such file), '/usr/local/lib/libpEpEngine.dylib' (no such file), '/usr/lib/libpEpEngine.dylib' (no such file) Abort trap: 6 ``` ### Expected Behaviour Build succeeds and dylib loads as usual ### Impact High - i need debugging symbols

I will try to check if it is something "obvious".

So, i compared the compile and linking lines between DEBUG= and DEBUG=debug and i could not find any other difference than:

DPEP_SAFETY_MODE=PEP_SAFETY_MODE_RELEASE vs. -DPEP_SAFETY_MODE=PEP_SAFETY_MODE_DEBUG
-DPEP_LOG_LEVEL_MAXIMUM=PEP_LOG_LEVEL_PRODUCTION vs. -DPEP_LOG_LEVEL_MAXIMUM=PEP_LOG_LEVEL_FUNCTION

Its very weird.

So, i compared the compile and linking lines between `DEBUG=` and `DEBUG=debug` and i could not find any other difference than: * DPEP_SAFETY_MODE=PEP_SAFETY_MODE_RELEASE vs. -DPEP_SAFETY_MODE=PEP_SAFETY_MODE_DEBUG * -DPEP_LOG_LEVEL_MAXIMUM=PEP_LOG_LEVEL_PRODUCTION vs. -DPEP_LOG_LEVEL_MAXIMUM=PEP_LOG_LEVEL_FUNCTION Its very weird.

@positron Also checked DEBUG=maintainer
same problem

@positron Also checked `DEBUG=maintainer` same problem

@positron building v3.2.0-RC19 i get the error in any DEBUG variant.
FYI: So far i was always using system default compiler:
Apple clang version 13.1.6 (clang-1316.0.21.2.5)

But can confirm it same behaviour using macports clang:
clang version 14.0.6

@positron building `v3.2.0-RC19` i get the error in any `DEBUG` variant. FYI: So far i was always using system default compiler: `Apple clang version 13.1.6 (clang-1316.0.21.2.5)` But can confirm it same behaviour using macports clang: `clang version 14.0.6`

@positron
v3.2.0-RC15 - config DEBUG= works using both compilers

@positron v3.2.0-RC15 - config `DEBUG=` works using both compilers

I am always using the same compiler for all code involved.

(sorry forgot to actually push 'send' thats why the comment is late)
@positron: so, magically enough, even though clearly not solved intentionallu, the engine v3.2.0-RC20 builds for me. (NOT suffering from #115)

If we dont fix #115 like NOW i fear its gonna be pain in the neck for the next 6 months that randomly pops up again.
And again am i right with the assumption that there is a clear plan changing the build to use sequoia statically?
because the last i know is a very VERY clear statement from volker that this problem WILL be around until we do that.
So if we want to be reasonable we have to either:

Revoke the statement (analysis)
or implement the solution
the reason i make it such a priority is whenever this problem strikes it is blocking in every situation

(sorry forgot to actually push 'send' thats why the comment is late) @positron: so, magically enough, even though clearly not solved intentionallu, the engine v3.2.0-RC20 builds for me. (NOT suffering from https://gitea.pep.foundation/pEp.foundation/pEpEngine/issues/115) If we dont fix #115 like NOW i fear its gonna be pain in the neck for the next 6 months that randomly pops up again. And again am i right with the assumption that there is a clear plan changing the build to use sequoia statically? because the last i know is a very VERY clear statement from volker that this problem WILL be around until we do that. So if we want to be reasonable we have to either: * Revoke the statement (analysis) * or implement the solution the reason i make it such a priority is whenever this problem strikes it is blocking in every situation

I am waiting for feedback from macos people here.

Now we are compiling the C part with slightly different and better-chosen flags, but this has not affected the problem.

I am waiting for feedback from macos people here. Now we are compiling the C part with slightly different and better-chosen flags, but this has not affected the problem.

@positron where/how can i test this?

[14:46] < heck> so, i am using v3.2-RC22 and the mac build problem #115 - BUILD: macOS build broken
[14:47] < heck> seems to be "solved" somehow
[14:47] < heck> so, lets forget it for now i would suppose, but i bet the phantom will appear again in the worst moment.. :/

[14:46] < heck> so, i am using v3.2-RC22 and the mac build problem https://gitea.pep.foundation/pEp.foundation/pEpEngine/issues/115 - BUILD: macOS build broken [14:47] < heck> seems to be "solved" somehow [14:47] < heck> so, lets forget it for now i would suppose, but i bet the phantom will appear again in the worst moment.. :/

Just out of the blue, this strikes again:

dyld[11434]: Library not loaded: 'libpEpEngine.dylib'
  Referenced from: '/Users/heck/src/pEp/libpEpAdapter32/test/test_adapter_cxx'
  Reason: tried: '/Users/heck/local-3x/lib/libpEpEngine.dylib' (code signature does not cover entire file up to signature),....

I know this is not pEpEngine's fault, but this is blocking and i dont know what to do, really.
I work on a variety of repos on this machine and never seen this before ever.

Just out of the blue, this strikes again: ``` dyld[11434]: Library not loaded: 'libpEpEngine.dylib' Referenced from: '/Users/heck/src/pEp/libpEpAdapter32/test/test_adapter_cxx' Reason: tried: '/Users/heck/local-3x/lib/libpEpEngine.dylib' (code signature does not cover entire file up to signature),.... ``` I know this is not pEpEngine's fault, but this is blocking and i dont know what to do, really. I work on a variety of repos on this machine and never seen this before ever.

looks like a reboot can help (not kidding), and a different installation path

https://developer.apple.com/forums/thread/700053

looks like a reboot can help (not kidding), and a different installation path https://developer.apple.com/forums/thread/700053

@vb-roar oh why did i not find this? thank a lot volker!

So, my understanding is:

seems to be only the case when using cc flag -g.
the kernel is caching signatures
file 'content' replaced -> signature mismatch compared with cache

Without having tried anything yet, i propose the following solution approach.

@heck removes -g flag in release builds.
@heck figures out options:

macOS only: clearing kernel-chache as part of the build
macOS only: add extra hoop jumps to the build, to get the kernel to update his cache

@positron as this is concerning macOS only, and you have no way to reproduce this, problems like this must like flying blindly and really annoying for you.
So, i propose i do a PR for all of that.

@vb-roar oh why did i not find this? thank a lot volker! So, my understanding is: * seems to be only the case when using `cc` flag `-g`. * the kernel is caching signatures * file 'content' replaced -> signature mismatch compared with cache Without having tried anything yet, i propose the following solution approach. 1) @heck removes `-g` flag in release builds. 2) @heck figures out options: * macOS only: clearing kernel-chache as part of the build * macOS only: add extra hoop jumps to the build, to get the kernel to update his cache @positron as this is concerning macOS only, and you have no way to reproduce this, problems like this must like flying blindly and really annoying for you. So, i propose i do a PR for all of that.

So, @positron,
The problem is still not solved, its striking back right now.

In reference to my suggested further actions in my last comment:
"seems to be only the case when using cc flag -g."

I did remove the -g flag in the release build.
It immediately solved the problem.

Could i please suggest the -g flag to be removed in DEBUG=release builds.
Then, at least in release builds the problem seems to be solved.

Thanks you.

So, @positron, The problem is still not solved, its striking back right now. In reference to my suggested further actions in my last comment: "seems to be only the case when using cc flag -g." I did remove the `-g` flag in the release build. It immediately solved the problem. Could i please suggest the `-g` flag to be removed in `DEBUG=release` builds. Then, at least in release builds the problem seems to be solved. Thanks you.

Could i please suggest the -g flag to be removed in DEBUG=release builds.
Then, at least in release builds the problem seems to be solved.

I included this suggested workaround in
https://gitea.pep.foundation/pEp.foundation/pEpEngine/releases/tag/v3.2.0-RC25
. I would prefer this trick to be temporary but it is not a problem to keep it as long as it is needed.

> Could i please suggest the `-g` flag to be removed in `DEBUG=release` builds. > Then, at least in release builds the problem seems to be solved. I included this suggested workaround in https://gitea.pep.foundation/pEp.foundation/pEpEngine/releases/tag/v3.2.0-RC25 . I would prefer this trick to be temporary but it is not a problem to keep it as long as it is needed.

@positron Thanks luca

yes, hopefully option 2 solves the problem in a better way

@heck figures out options:

    macOS only: clearing kernel-chache as part of the build
    macOS only: add extra hoop jumps to the build, to get the kernel to update his cache

@positron Thanks luca yes, hopefully option 2 solves the problem in a better way ``` @heck figures out options: macOS only: clearing kernel-chache as part of the build macOS only: add extra hoop jumps to the build, to get the kernel to update his cache ```

Labels Milestones

BUILD: macOS build broken #115

Reproduction

Observed Behaviour

Expected Behaviour

Impact