chore(release): 0.4.3

fix(big-sur): resolve issues with building and native-comp on Big Sur

Brewfile

@@ -3,11 +3,13 @@
 brew 'coreutils'
 brew 'curl'
 brew 'expat'
+brew 'gcc'
 brew 'gmp'
 brew 'gnu-sed'
 brew 'gnutls'
 brew 'jansson'
 brew 'libffi'
+brew 'libgccjit'
 brew 'libiconv'
 brew 'librsvg'
 brew 'libtasn1'

CHANGELOG.md

@@ -2,6 +2,40 @@
 
 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
+### [0.4.3](https://github.com/jimeh/build-emacs-for-macos/compare/0.4.2...0.4.3) (2020-12-28)
+
+
+### Bug Fixes
+
+* **big-sur:** add Xcode CLI tools lib directory to runtime LIBRARY_PATH ([946856e](https://github.com/jimeh/build-emacs-for-macos/commit/946856e9c242d4a6fb5f839d8cae0acfafecdfc6))
+* **big-sur:** added support for building on Big Sur ([2247158](https://github.com/jimeh/build-emacs-for-macos/commit/2247158051d0f59933569b6974b2b5269f13c79e))
+
+### [0.4.2](https://github.com/jimeh/build-emacs-for-macos/compare/0.4.1...0.4.2) (2020-12-09)
+
+
+### Bug Fixes
+
+* **cli:** avoid error if --git-sha is used without a branch/tag/sha argument ([884f160](https://github.com/jimeh/build-emacs-for-macos/commit/884f1607f6707ca187b1abfb0ce562757d872230)), closes [#21](https://github.com/jimeh/build-emacs-for-macos/issues/21)
+* **native_comp:** update env setup patch for recent changes to comp.el ([c7daa13](https://github.com/jimeh/build-emacs-for-macos/commit/c7daa1350bd69df172ce6484c54189d2cee8d97e))
+
+### [0.4.1](https://github.com/jimeh/build-emacs-for-macos/compare/0.4.0...0.4.1) (2020-10-29)
+
+
+### Features
+
+* **native_comp:** remove patch based on feature/native-comp-macos-fixes branch ([70bf6b0](https://github.com/jimeh/build-emacs-for-macos/commit/70bf6b05d584976632b2fd2947c0bf692f5b6421))
+
+## [0.4.0](https://github.com/jimeh/build-emacs-for-macos/compare/0.3.0...0.4.0) (2020-10-04)
+
+
+### ⚠ BREAKING CHANGES
+
+* **native_comp:** Standard Homewbrew `gcc` and `libgccjit` formula are now required for native-comp, instead of the custom patched gcc formula.
+
+### Features
+
+* **native_comp:** use new libgccjit Homebrew formula ([d8bbcb7](https://github.com/jimeh/build-emacs-for-macos/commit/d8bbcb72b33f6bde8678c9d37548217ffdf3d641))
+
 ## [0.3.0](https://github.com/jimeh/build-emacs-for-macos/compare/0.2.0...0.3.0) (2020-09-22)
 
 

Formula/gcc.rb.patch

@@ -1,21 +0,0 @@
-diff --git a/Formula/gcc.rb b/Formula/gcc.rb
-index 1bd636d496..03ad124218 100644
---- a/Formula/gcc.rb
-+++ b/Formula/gcc.rb
-@@ -53,7 +53,7 @@ class Gcc < Formula
-     #  - Ada, which requires a pre-existing GCC Ada compiler to bootstrap
-     #  - Go, currently not supported on macOS
-     #  - BRIG
--    languages = %w[c c++ objc obj-c++ fortran]
-+    languages = %w[c c++ objc obj-c++ fortran jit]
-
-     osmajor = `uname -r`.split(".").first
-     pkgversion = "Homebrew GCC #{pkg_version} #{build.used_options*" "}".strip
-@@ -73,6 +73,7 @@ class Gcc < Formula
-       --with-system-zlib
-       --with-pkgversion=#{pkgversion}
-       --with-bugurl=https://github.com/Homebrew/homebrew-core/issues
-+      --enable-host-shared
-     ]
-
-     # Xcode 10 dropped 32-bit support

README.md

@@ -18,7 +18,7 @@ Use this script at your own risk.
 
 ## Status
 
-As of writing (2020-08-19) it works for me on my machine. Your luck may vary.
+As of writing (2020-10-04) it works for me on my machine. Your luck may vary.
 
 I have successfully built:
 
@@ -30,7 +30,7 @@ For reference, my machine is:
 
 - 13-inch MacBook Pro (2020), 10th-gen 2.3 GHz Quad-Core Intel Core i7 (4c/8t)
 - macOS 10.15.6 (19G2021)
-- Xcode 11.7
+- Xcode 12.0
 
 ## Limitations
 
@@ -76,11 +76,11 @@ Options:
         --[no-]native-full-aot       Enable/disable NATIVE_FULL_AOT / Ahead of Time compilation (default: disabled)
         --[no-]native-comp-macos-fixes
                                      Enable/disable fix based on feature/native-comp-macos-fixes branch (default: enabled if native-comp supported)
-        --[no-]launcher              Enable/disable embedded launcher script  (default: enabled if native-comp is enabled)
         --rsvg                       Enable SVG image support via librsvg, can yield a unstable build (default: disabled)
         --no-titlebar                Apply no-titlebar patch (default: disabled)
         --no-frame-refocus           Apply no-frame-refocus patch (default: disabled)
         --[no-]native-fast-boot      DEPRECATED: use --[no-]native-full-aot instead
+        --[no-]launcher              DEPRECATED: Launcher script is no longer used.
 ```
 
 Resulting applications are saved to the `builds` directory in a bzip2 compressed
@@ -89,6 +89,9 @@ tarball.
 If you don't want the build process to eat all your CPU cores, pass in a `-j`
 value of how many CPU cores you want it to use.
 
+Re-building the same Git SHA again can yield weird results unless you first
+trash the corresponding directory from the `sources` directory.
+
 ### Examples
 
 To download a tarball of the `master` branch (Emacs 28.x as of writing) and
@@ -109,31 +112,42 @@ All sources as downloaded as tarballs from the
 to get a list of tags/branches available to install, simply check said
 repository.
 
+## Use Self-Contained Emacs.app as `emacs` CLI Tool
+
+As the application bundle is self-contained, the main executable needs to be run
+from within the application bundle. This means a simple symlink to
+`Emacs.app/Contents/MacOS/Emacs` will not work. Instead the best approach is to
+create a shell alias called `emacs` pointing to the right place.
+
+Personally I use something similar to this:
+
+```bash
+if [ -f "/Applications/Emacs.app/Contents/MacOS/Emacs" ]; then
+  export EMACS="/Applications/Emacs.app/Contents/MacOS/Emacs"
+  alias emacs="$EMACS -nw"
+fi
+
+if [ -f "/Applications/Emacs.app/Contents/MacOS/bin/emacsclient" ]; then
+  alias emacsclient="/Applications/Emacs.app/Contents/MacOS/bin/emacsclient"
+fi
+```
+
+Setting the `EMACS` variable to the binary path seems to be a good idea, as some
+tools seems to use it to figure out the path to Emacs' executable, including
+[doom-emacs](https://github.com/hlissner/doom-emacs)' `doom` CLI tool.
+
 ## Native-Comp
 
-To build a Emacs.app with native-comp support
+Building a Emacs.app with native-comp support
 ([gccemacs](https://akrl.sdf.org/gccemacs.html)) from the `feature/native-comp`
-branch, you will need to install a patched version of Homebrew's `gcc` formula
+branch is now supported without much hassle thanks to the newly released
-that includes libgccjit.
+`libgccjit` Homebrew formula.
 
-The patch itself is in `./Formula/gcc.rb.patch`, and comes from
+Changes from the `feature/native-comp-macos-fixes` branch are also applied
-[this](https://gist.github.com/mikroskeem/0a5c909c1880408adf732ceba6d3f9ab#1-gcc-with-libgccjit-enabled)
+through a custom patch process which should be more future-proof compared to a
-gist.
+regular git diff patch.
 
-You can install the patched formula by running the helper script:
+To build a Emacs.app with native compilation enabled, simply run:
-
-```
-./install-patched-gcc
-```
-
-The helper script will copy your local `gcc.rb` Forumla from Homebrew to
-`./Formula`, and apply the `./Formula/gcc.rb.patch` to it. After which it then
-proceed to install the patched gcc formula which includes libgccjit.
-
-As it requires installing and compiling GCC from source, it can take anywhere
-between 30-60 minutes or more depending on your machine.
-
-And finally to build a Emacs.app with native compilation enabled, run:
 
 ```
 ./build-emacs-for-macos feature/native-comp
@@ -142,7 +156,7 @@ And finally to build a Emacs.app with native compilation enabled, run:
 By default `NATIVE_FULL_AOT` is disabled which ensures a fast build by native
 compiling as few lisp source files as possible to build the app. Any remaining
 lisp files will be dynamically compiled in the background the first time you use
-them.
+them. To enable native full AoT, pass in the `--native-full-aot` option.
 
 On my machine it takes around 10 minutes to build Emacs.app with
 `NATIVE_FULL_AOT` disabled. With it enabled it takes around 20-25 minutes.

build-emacs-for-macos

@@ -78,7 +78,7 @@ class Build
   attr_reader :ref
   attr_reader :options
 
-  def initialize(root_dir, ref = 'master', options = {})
+  def initialize(root_dir, ref = nil, options = {})
     @root_dir = root_dir
     @ref = ref
     @options = options
@@ -99,7 +99,7 @@ class Build
     symlink_internals(app)
 
     LibEmbedder.new(app, brew_dir, extra_libs).embed
-    LibGccJitEmbedder.new(app, gcc_dir).embed if options[:native_comp]
+    GccLibEmbedder.new(app, gcc_dir).embed if options[:native_comp]
 
     archive_app(app)
   end
@@ -126,6 +126,10 @@ class Build
     @gcc_dir ||= `brew --prefix gcc`.chomp
   end
 
+  def libgccjit_dir
+    @libgccjit_dir ||= `brew --prefix libgccjit`.chomp
+  end
+
   def extra_libs
     @extra_libs ||= [
       "#{brew_dir}/opt/expat/lib/libexpat.1.dylib",
@@ -139,7 +143,7 @@ class Build
 
     url = (DOWNLOAD_URL % sha)
     filename = "emacs-mirror-emacs-#{sha[0..6]}.tgz"
-    target   = File.join(tarballs_dir, filename)
+    target = File.join(tarballs_dir, filename)
 
     if File.exist?(target)
       info "#{filename} already exists locally, attempting to use."
@@ -204,17 +208,25 @@ class Build
 
   def verify_libgccjit
     err 'gcc not installed' unless Dir.exist?(gcc_dir)
+    err 'libgccjit not installed' unless Dir.exist?(libgccjit_dir)
 
-    return if Dir["#{gcc_dir}/lib/**/libgccjit.so*"].any?
+    if Dir["#{libgccjit_dir}/lib/**/libgccjit.so*"].empty?
+      err "Detected libgccjit (#{libgccjit_dir}) does not have any " \
+          'libgccjit.so* files. Please try reinstalling libgccjit: ' \
+          'brew reinstall libgccjit'
+    end
 
-    err "Detected GCC (#{gcc_dir}) does not have libgccjit. Ensure patched " \
+    dirs = Dir["{#{gcc_dir},#{libgccjit_dir}}/lib/gcc/*"]
-        'gcc brew formula has been installed via ./install-patched-gcc'
+    return if dirs.map { |dir| File.basename(dir) }.uniq == %w[10]
+
+    err 'Detected gcc and libgccjit library paths do not belong to the ' \
+        "same major version. Detected paths:\n  - #{dirs.join('  - ')}"
   end
 
   def gcc_library_paths
     @gcc_library_paths ||= Dir[
-      "#{gcc_dir}/lib/gcc/*",
+      "{#{gcc_dir},#{libgccjit_dir}}/lib/gcc/*",
-      "#{gcc_dir}/lib/gcc/*/gcc/*apple-darwin*/*"
+      "{#{gcc_dir},#{libgccjit_dir}}/lib/gcc/*/gcc/*apple-darwin*/*"
     ].sort_by { |p| [p.size, p] }
   end
 
@@ -246,10 +258,6 @@ class Build
         verify_native_comp
         verify_libgccjit
 
-        if options[:macos_fixes] && ref != 'feature/native-comp-macos-fixes'
-          apply_native_comp_macos_fixes
-        end
-
         apply_native_comp_env_setup_patch(source)
 
         ENV['CFLAGS'] = [
@@ -260,7 +268,8 @@ class Build
 
         ENV['LDFLAGS'] = [
           gcc_library_paths.map { |path| "-L#{path}" },
-          "-I#{gcc_dir}/include"
+          "-I#{gcc_dir}/include",
+          "-I#{libgccjit_dir}/include"
         ].flatten.compact.join(' ')
 
         ENV['LIBRARY_PATH'] = [
@@ -289,6 +298,11 @@ class Build
         "#{brew_dir}/opt/texinfo/bin",
         ENV['PATH']
       ].compact.join(':')
+      
+      ENV['LIBRARY_PATH'] = [
+         ENV['LIBRARY_PATH'],
+         "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib"
+        ].compact.join(':')
 
       configure_flags = [
         '--with-ns',
@@ -357,18 +371,18 @@ class Build
     FileUtils.mkdir_p(builds_dir)
 
     metadata = [
-      ref.gsub(/\W/, '-'),
+      meta[:ref]&.gsub(/\W/, '-'),
       meta[:date],
       meta[:sha][0..6],
       "macOS-#{OS.version}",
       OS.arch
-    ]
+    ].compact
 
     filename = "Emacs.app-[#{metadata.join('][')}].tbz"
-    target   = "#{builds_dir}/#{filename}"
+    target = "#{builds_dir}/#{filename}"
 
     app_base = File.basename(app)
-    app_dir  = File.dirname(app)
+    app_dir = File.dirname(app)
 
     if !File.exist?(target)
       info "Creating #{filename} archive in \"#{builds_dir}\"..."
@@ -403,11 +417,13 @@ class Build
     err "Failed to get commit info about: #{ref_sha}" if commit_json.nil?
 
     commit = JSON.parse(commit_json)
-
+    meta = {
-    @meta = {
       sha: commit['sha'],
       date: Date.parse(commit['commit']['committer']['date'])
     }
+    meta[:ref] = ref if ref && ref[0..6] != meta[:sha][0..6]
+
+    @meta = meta
   end
 
   def http_get(url)
@@ -433,37 +449,6 @@ class Build
     )
   end
 
-  def apply_native_comp_macos_fixes
-    filename = 'Makefile.in'
-    pattern = /^src: Makefile\n(.*BIN_DESTDIR.*)\nblessmail: Makefile src\n/m
-    content = File.read(filename).gsub(pattern) do
-      old_src_body = Regexp.last_match(1).strip
-
-      # check if already patched
-      if old_src_body.include?('BIN_DESTDIR=\'${ns_appbindir}/\'')
-        return old_src_body
-      end
-
-      self_contained = old_src_body.gsub(
-        'BIN_DESTDIR=\'$(DESTDIR)${bindir}/\'',
-        'BIN_DESTDIR=\'${ns_appbindir}/\''
-      )
-
-      <<~REPLACEMENT
-        src: Makefile
-        ifeq (${ns_self_contained},no)
-        \t#{old_src_body}
-        else
-        \t#{self_contained}
-        endif
-
-        blessmail: Makefile src
-      REPLACEMENT
-    end
-
-    File.open(filename, 'w') { |f| f.write(content) }
-  end
-
   def effective_version
     @effective_version ||= begin
       case ref
@@ -587,7 +572,7 @@ class AbstractEmbedder
   end
 
   def lib_dir_name
-    "lib-#{OS.arch}-#{OS.version.to_s.tr('.', '_')}"
+    'lib'
   end
 end
 
@@ -700,7 +685,7 @@ class LibEmbedder < AbstractEmbedder
   end
 end
 
-class LibGccJitEmbedder < AbstractEmbedder
+class GccLibEmbedder < AbstractEmbedder
   attr_reader :gcc_dir
 
   def initialize(app, gcc_dir)
@@ -721,6 +706,7 @@ class LibGccJitEmbedder < AbstractEmbedder
 
     FileUtils.mkdir_p(File.dirname(target_dir))
     FileUtils.cp_r(source_dir, target_dir)
+    FileUtils.rm(Dir[File.join(target_dir, '**', '.DS_Store')], force: true)
   end
 
   private
@@ -734,13 +720,10 @@ class LibGccJitEmbedder < AbstractEmbedder
   end
 
   def gcc_version
-    @gcc_version ||= Dir[File.join(gcc_dir, 'lib', 'gcc', '*', 'libgccjit.so*')]
+    @gcc_version ||= Dir[File.join(gcc_dir, 'lib', 'gcc', '*', 'libgcc*')]
-                     .map { |path| File.dirname(path) }
+                     .map { |path| File.basename(File.dirname(path)) }
-                     .select { |path| path.match(%r{/\d+$}) }
+                     .select { |path| path.match(/^\d+$/) }
-                     .uniq
+                     .uniq.map(&:to_i).max.to_s
-                     .map { |dir| File.basename(dir).to_i }
-                     .max
-                     .to_s
   end
 
   def source_dir

install-patched-gcc

@@ -1,27 +0,0 @@
-#! /usr/bin/env bash
-set -e
-
-brewdir="$(brew --prefix)"
-formula="${brewdir}/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/gcc.rb"
-
-if [ ! -f "$formula" ]; then
-  echo "ERROR: ${formula} does not exist." 1>&2
-  exit 1
-fi
-
-gnubins=(
-  "${brewdir}/opt/coreutils/libexec/gnubin"
-  "${brewdir}/opt/make/libexec/gnubin"
-  "${brewdir}/opt/gnu-sed/libexec/gnubin"
-)
-
-for gnubin in "${gnubins[@]}"; do
-  if [ -d "$gnubin" ]; then
-    export PATH="${gnubin}:$PATH"
-  fi
-done
-
-cp "$formula" ./Formula/
-
-patch -f -p1 -i ./Formula/gcc.rb.patch
-brew install ./Formula/gcc.rb --build-from-source --force

patches/native-comp-env-setup.patch

@@ -1,8 +1,8 @@
 diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
-index 25e2de9..14ee6dc 100644
+index 25e2de9..bcedd31 100644
 --- a/lisp/emacs-lisp/comp.el
 +++ b/lisp/emacs-lisp/comp.el
-@@ -2801,6 +2801,36 @@ queued with LOAD %"
+@@ -2801,6 +2801,57 @@ queued with LOAD %"
        (comp-run-async-workers)
        (message "Compilation started."))))
  
@@ -13,28 +13,49 @@ index 25e2de9..14ee6dc 100644
 +             (string-match-p "\.app\/Contents\/MacOS\/?$"
 +                             invocation-directory))
 +    (let* ((library-path-env (getenv "LIBRARY_PATH"))
-+           (gcc-dir (car (file-expand-wildcards
++           (devtools-dir
-+                          (concat invocation-directory "lib-*/gcc/*"))))
++            "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib")
-+           (gcc-darwin-dir (car (file-expand-wildcards
++           (gcc-base-dir (concat invocation-directory "lib/gcc"))
-+                                 (concat gcc-dir "/gcc/*apple-darwin*/*"))))
++           (gcc-version (car (seq-filter
++                              (lambda (dir) (string-match-p "^[0-9]+$" dir))
++                              (directory-files gcc-base-dir))))
++           (gcc-dir (concat gcc-base-dir "/" gcc-version))
++           (darwin-base-dir (car (file-expand-wildcards
++                                      (concat gcc-dir "/gcc/*apple-darwin*"))))
++           (darwin-version (car (seq-filter
++                                 (lambda (dir)
++                                   (string-match-p
++                                    "^[0-9]+\\(\.[0-9]+\\(\.[0-9]+\\)?\\)?$" dir))
++                                 (directory-files darwin-base-dir))))
++           (darwin-dir (concat darwin-base-dir "/" darwin-version))
 +           (lib-paths (append
-+                       (list gcc-dir gcc-darwin-dir)
++                       (list gcc-dir darwin-dir)
++                       (if (file-directory-p devtools-dir)
++                           (list devtools-dir) (list))
 +                       (if library-path-env (list library-path-env) (list)))))
 +
-+      (when (and gcc-dir gcc-darwin-dir)
++      (when (and gcc-dir darwin-dir)
 +        (setenv "LIBRARY_PATH" (mapconcat 'identity lib-paths ":")))))
 +
 +  ;; Remove advice, as it only needs to run once.
 +  (advice-remove 'native-compile
 +                 'native-compile-setup-environment-variables)
++  (advice-remove 'comp--native-compile
++                 'native-compile-setup-environment-variables)
 +  (advice-remove 'native-compile-async
++                 'native-compile-setup-environment-variables)
++  (advice-remove 'native--compile-async
 +                 'native-compile-setup-environment-variables))
 +
 +;; Ensure environment setup runs before any native compilation.
 +(advice-add 'native-compile :before
 +            'native-compile-setup-environment-variables)
++(advice-add 'comp--native-compile :before
++            'native-compile-setup-environment-variables)
 +(advice-add 'native-compile-async :before
 +            'native-compile-setup-environment-variables)
++(advice-add 'native--compile-async :before
++            'native-compile-setup-environment-variables)
 +
  (provide 'comp)