Enable building Emacs 27

Also embed various dylib libraries directly into the application, so
they're not required to be installed via Homebrew for the app to work.
This commit is contained in:
2020-02-02 17:11:13 +00:00
parent 476ac73528
commit 7986e89623
5 changed files with 490 additions and 284 deletions

15
Brewfile Normal file
View File

@@ -0,0 +1,15 @@
# frozen_string_literal: true
brew 'gmp'
brew 'gnutls'
brew 'jansson'
brew 'libffi'
brew 'libiconv'
brew 'libtasn1'
brew 'libunistring'
brew 'libxml2'
brew 'ncurses'
brew 'nettle'
brew 'pkg-config'
brew 'texinfo'
brew 'zlib'

View File

@@ -1,3 +1,8 @@
# frozen_string_literal: true
source 'http://rubygems.org/'
gem 'json', :platform => :ruby_18
group :development do
gem 'byebug'
gem 'rubocop'
end

View File

@@ -1,63 +1,77 @@
# build-emacs-for-osx
# build-emacs-for-macos
Use this script at your own risk. It currently works for me on my own machine,
which as of writing is:
Use this script at your own risk. As of writing (2020-02-02) it works for me on
my own machine to build the `emacs-27` release branch. My machine is a late-2016
13-inch Touchbar MacBook Pro runnning macOS 10.15.2 and Xcode 11.3.
* OS X 10.8.5 (12F45)
* Xcode 5.0 (5A1413)
Your luck may vary.
Your luck might vary. Do note that it does not build a universal application.
The CPU architecture of the built application will be that of the machine it
was built on.
The build produced does have some limitations:
- It is not a universal application. The CPU architecture of the built
application will be that of the machine it was built on.
- The minimum required macOS version of the built application will be the same
as that of the machine it was built on.
## Why?
I've been using [Homebrew][] the past few
months to build from HEAD. Homebrew comes with a number of patches, including
the [sRGB][] patches which I use.
- To use new features available from master or pre-release branches, which have
not made it into a official stable release yet.
- Homebrew builds of Emacs are not self-contained applications, making it very
difficult when doing HEAD builds and you need to rollback to a earlier
version.
- Builds from [emacsformacosx.com](https://emacsformacosx.com/) has had no new
nightly builds for two months right now.
- Both Homebrew HEAD builds, and nightly builds from emacsformacosx.com are
built from the `master` branch. This script allows you to choose any branch
you want. I am currently building from the `emacs-27` branch which is the
basis of the upcoming Emacs 27 release, meaning it should be more stable than
`master` builds.
[homebrew]: http://mxcl.github.com/homebrew/
[srgb]: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8402
Homebrew does not build a self-contained application though, which caused
issues for me when I needed to rollback to a specific build. I found the
easiest way to build a completely self-contained Emacs.app nightly from a
specific date with custom patches was to do it manually.
So I decided to quickly hack together a script to automate that manual
process. The code is a horrible hack, but it (seemingly) works as I'm writing
this in Emacs built with it.
## Requirements
- [Xcode](https://apps.apple.com/gb/app/xcode/id497799835?mt=12)
- [Homebrew](https://brew.sh/)
- All Homebrew formula listed in the `Brewfile`, which can all easily be
installed by running:
```
brew bundle
```
## Usage
Myself I run the following command which will download a tarball of the
`master` branch, apply the sRGB patch, and build Emacs.app:
Then to download a tarball of the `master` branch, build Emacs.app:
./build-emacs-for-osx
./build-emacs-for-macos
Or for example if you want to build the `emacs-24.3` tag, run:
If you want to build the `emacs-27` git branch, run:
./build-emacs-for-osx emacs-24.3
./build-emacs-for-macos emacs-27
Resulting applications are saved to the `builds` directory in a bzip2
compressed tarball.
If you want to build the stable `emacs-26.3` git tag, run:
./build-emacs-for-macos emacs-26.3
Resulting applications are saved to the `builds` directory in a bzip2 compressed
tarball.
## Internals
I decided to pull Emacs' source from a GitHub [mirror][repo] rather than the
official Bzr repo cause I'm not familiar with Bzr, and GitHub lets you easily
download tarballs of any branch, commit or tag. And the tarballs from GitHub
are just over 30MB, compared to ~1GB to pull the offical Bzr repo.
The script downloads the source code as a gzipped tar archive from the [GitHub
mirror](https://github.com/emacs-mirror/emacs) repository, as it makes it very
easy to get a tarball of any given git reference.
[repo]: https://github.com/mirrors/emacs
The only option passed in `./configure` is `--with-ns`, meaning the resulting
application only supports the CPU architecture of the system is was built on.
There might be more side-effects to, but I haven't noticed any.
It then runs `./configure` with a various options, partly based on what [David
Caldwell](https://github.com/caldwell)'s
[build-emacs](https://github.com/caldwell/build-emacs) scripts do, including
copying various dynamic libraries into the application itself. So the built
application should in theory run on a macOS install that does not have homebrew,
or do no have the relevant brew formula installed.
Code quality, is well, non-existent. The build script started life a super-quick
hack back in 2013, and now it's even more of a dirty hack. I might clean it up
and add unit tests if I end up relying on this script for a prolonged period of
time. For now I plan to use it until Emacs 27 is officially released.
## License
@@ -65,7 +79,7 @@ There might be more side-effects to, but I haven't noticed any.
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 2, December 2004
Copyright (C) 2013 Jim Myhrberg
Copyright (C) 2020 Jim Myhrberg
Everyone is permitted to copy and distribute verbatim or modified
copies of this license document, and changing it is allowed as long

416
build-emacs-for-macos Executable file
View File

@@ -0,0 +1,416 @@
#!/usr/bin/env ruby
# frozen_string_literal: true
require 'date'
require 'fileutils'
require 'json'
require 'optparse'
require 'pathname'
#
# Config
#
DOWNLOAD_URL = 'https://github.com/emacs-mirror/emacs/tarball/%s'
LATEST_URL = 'https://api.github.com/repos/emacs-mirror/emacs/commits?sha=%s'
ROOT_DIR = File.expand_path(__dir__)
TARBALL_DIR = "#{ROOT_DIR}/tarballs"
SOURCES_DIR = "#{ROOT_DIR}/sources"
BUILDS_DIR = "#{ROOT_DIR}/builds"
#
# Main
#
def main
opts = parse_options
ref = ARGV.shift
meta = get_ref_info(ref)
if meta['sha'] && meta['date']
tarball = download_tarball(meta['sha'])
source = extract_tarball(tarball, patches(opts))
app = compile_source(source, opts)
internalize_libraries(app)
archive_app(app, ref, meta['sha'], meta['date'])
else
raise "\nERROR: Failed to get commit info from GitHub API."
end
end
#
# Patches
#
def patches(opts = {})
p = []
if opts[:xwidgets]
p << {
url: 'https://gist.github.com/fuxialexander/' \
'0231e994fd27be6dd87db60339238813/raw/' \
'b30c2d3294835f41e2c8afa1e63571531a38f3cf/0_all_webkit.patch'
}
end
p
end
#
# Options
#
def parse_options
options = {}
OptionParser.new do |opts|
opts.banner = <<~DOC
Usage: ./build-emacs-for-macos [options] <branch/tag/sha>
Branch, tag, and SHA are from the mirrors/emacs Github repo,
available here: https://github.com/mirrors/emacs
DOC
opts.on('-j', '--parallel PROCS',
'Compile in parallel using PROCS processes') do |v|
options[:parallel] = v
end
opts.on('-x', '--xwidgets', 'Compile in parallel using PROCS processes') do
options[:xwidgets] = true
end
end.parse!
options
end
#
# Core Methods
#
def download_tarball(sha)
FileUtils.mkdir_p(TARBALL_DIR)
url = (DOWNLOAD_URL % sha)
filename = "emacs-mirror-emacs-#{sha[0..6]}.tgz"
target = File.join(TARBALL_DIR, filename)
if !File.exist?(target)
puts "\nDownloading tarball from GitHub. This could take a while, " \
'please be patient.'
unless run_cmd("curl -L \"#{url}\" -o \"#{target}\"")
raise "\nERROR: Download failed."
end
else
puts "\nINFO: #{filename} already exists locally, attempting to use."
end
target
end
def extract_tarball(filename, patches = [])
FileUtils.mkdir_p(SOURCES_DIR)
dirname = File.basename(filename).gsub(/\.\w+$/, '')
target = "#{SOURCES_DIR}/#{dirname}"
if !File.exist?(target)
puts "\nExtracting tarball..."
unless run_cmd("tar -xzf \"#{filename}\" -C \"#{SOURCES_DIR}\"")
raise "\nERROR: Tarball extraction failed."
end
patches.each do |patch|
apply_patch(patch, target)
end
else
puts "\nINFO: #{dirname} source tree exists, attempting to use."
end
target
end
def compile_source(source, opts)
target = "#{source}/nextstep"
if !File.exist?("#{target}/Emacs.app")
puts "\nCompiling from source. This will take a while..."
v = get_macos_version
ver = "#{v[:major]}.#{v[:minor]}"
brew_dir = `brew --prefix`.chomp
ENV['CC'] = 'cc'
ENV['PKG_CONFIG_PATH'] = [
"#{brew_dir}/lib/pkgconfig",
"#{brew_dir}/share/pkgconfig",
"#{brew_dir}/opt/expat/lib/pkgconfig",
"#{brew_dir}/opt/libxml2/lib/pkgconfig",
"#{brew_dir}/opt/ncurses/lib/pkgconfig",
"#{brew_dir}/opt/zlib/lib/pkgconfig",
"#{brew_dir}/Homebrew/Library/Homebrew/os/mac/pkgconfig/#{ver}",
ENV['PKG_CONFIG_PATH']
].compact.join(':')
ENV['PATH'] = [
"#{brew_dir}/bin",
"#{brew_dir}/opt/texinfo/bin",
ENV['PATH']
].compact.join(':')
configure_flags = [
'--with-ns',
'--with-modules',
'--enable-locallisppath=' \
'/Library/Application Support/Emacs/${version}/site-lisp:' \
'/Library/Application Support/Emacs/site-lisp'
]
configure_flags << '--with-xwidgets' if opts[:xwidgets]
parallel_flags = opts[:parallel] ? ['-j', opts[:parallel]] : []
FileUtils.cd(source) do
if File.exist?('autogen/copy_autogen')
run_cmd 'autogen/copy_autogen'
elsif File.exist?('autogen.sh')
run_cmd './autogen.sh'
end
run_cmd './configure', *configure_flags
# Disable aligned_alloc on Mojave and below. See issue:
# https://github.com/daviderestivo/homebrew-emacs-head/issues/15
if v[:major] <= 10 && v[:minor] <= 14
puts 'Force disabling of aligned_alloc on macOS <= Mojave (10.14.x)'
disable_alligned_alloc
end
run_cmd 'make', *parallel_flags
run_cmd 'make', 'install'
end
raise "\nERROR: Build failed." unless File.exist?("#{target}/Emacs.app")
else
puts "\nINFO: Emacs.app already exists in " \
"\"#{target.gsub(ROOT_DIR + '/', '')}\", attempting to use."
end
"#{target}/Emacs.app"
end
def internalize_libraries(app)
raise "\nERROR: #{app} does not exist" unless File.exist?(app)
puts "\nEmbedding libraries into Emacs.app"
v = get_macos_version
brew_dir = `brew --prefix`.chomp
lib_dir = "lib-x86_64-#{[v[:major], v[:minor]].join('_')}"
extra_libs = [
"#{brew_dir}/opt/expat/lib/libexpat.1.dylib",
"#{brew_dir}/opt/libiconv/lib/libiconv.2.dylib",
"#{brew_dir}/opt/zlib/lib/libz.1.dylib"
]
FileUtils.cd(File.dirname(app)) do
copy_libs(
"#{app}/Contents/MacOS/Emacs",
brew_dir,
"#{app}/Contents/MacOS/#{lib_dir}"
)
copy_extra_libs(
extra_libs,
"#{app}/Contents/MacOS/Emacs",
brew_dir,
"#{app}/Contents/MacOS/#{lib_dir}"
)
self_ref_libs(
"#{app}/Contents/MacOS/Emacs",
"#{app}/Contents/MacOS/#{lib_dir}"
)
end
end
def archive_app(app, ref, sha, date)
FileUtils.mkdir_p(BUILDS_DIR)
v = get_macos_version
metadata = [ref, date, sha[0..6], "macOS-#{v[:major]}.#{v[:minor]}"]
filename = "Emacs.app-[#{metadata.join('][')}].tbz"
target = "#{BUILDS_DIR}/#{filename}"
app_base = File.basename(app)
app_dir = File.dirname(app)
if !File.exist?(target)
puts "\nCreating #{filename} archive in \"#{BUILDS_DIR}\"..."
FileUtils.cd(app_dir) { system('tar', '-cjf', target, app_base) }
else
puts "\nINFO: #{filename} archive exists in " \
"#{BUILDS_DIR.gsub(ROOT_DIR + '/', '')}, skipping archving."
end
end
#
# Helper Methods
#
def run_cmd(*args)
puts '==> ' + args.join(' ')
system(*args)
end
def copy_libs(exe, brew_dir, lib_dir, rel_path = nil)
exe_file = File.basename(exe)
rel_path ||= Pathname.new(lib_dir).relative_path_from(File.dirname(exe)).to_s
`otool -L "#{exe}"`.split("\n")[1..-1].each do |line|
match = line.match(%r{^\s+(.+/(lib[^/ ]+))\s})
next unless match && match[1].start_with?(brew_dir)
while_writable(exe) do
if match[2] == exe_file
system('install_name_tool', '-id',
"@executable_path/#{rel_path}/#{match[2]}", exe)
else
system('install_name_tool', '-change', match[1],
"@executable_path/#{rel_path}/#{match[2]}", exe)
end
end
next if match[2] == exe_file || File.exist?("#{lib_dir}/#{match[2]}")
FileUtils.mkdir_p(lib_dir)
FileUtils.cp(match[1], lib_dir)
copy_libs("#{lib_dir}/#{match[2]}", brew_dir, lib_dir, rel_path)
end
end
def copy_extra_libs(extra_libs, exe, brew_dir, lib_dir, rel_path = nil)
rel_path ||= Pathname.new(lib_dir).relative_path_from(File.dirname(exe)).to_s
extra_libs.each do |lib|
lib_file = File.basename(lib)
target = "#{lib_dir}/#{lib_file}"
unless File.exist?(target)
FileUtils.mkdir_p(lib_dir)
FileUtils.cp(lib, lib_dir)
end
while_writable(target) do
system('install_name_tool', '-id',
"@executable_path/#{rel_path}/#{lib_file}", target)
end
copy_libs(target, brew_dir, lib_dir, rel_path)
end
end
def self_ref_libs(exe, lib_dir)
rel_path = Pathname.new(lib_dir).relative_path_from(File.dirname(exe)).to_s
lib_paths ||= Dir.glob("#{lib_dir}/*")
libs = lib_paths.map { |f| File.basename(f) }
([exe] + lib_paths).each do |bin_path|
`otool -L "#{bin_path}"`.split("\n")[1..-1].each do |line|
match = line.match(%r{^\s+(.+/(lib[^/ ]+))\s})
next unless match
next if match[1].start_with?('@executable_path/')
next unless libs.include?(match[2])
while_writable(bin_path) do
system('install_name_tool', '-change', match[1],
"@executable_path/#{rel_path}/#{match[2]}",
bin_path)
end
end
end
end
def while_writable(file)
mode = File.stat(file).mode
File.chmod(0o775, file)
yield
File.chmod(mode, file)
end
def get_macos_version
v = `sw_vers -productVersion`.chomp
.sub(/^(\d+\.\d+\.\d)+/, '\1')
.split('.')
.map(&:to_i)
{ major: v[0], minor: v[1], patch: v[2] }
end
def disable_alligned_alloc
filename = 'src/config.h'
content = File.read(filename)
.gsub('#define HAVE_ALIGNED_ALLOC 1',
'#undef HAVE_ALIGNED_ALLOC')
.gsub('#define HAVE_DECL_ALIGNED_ALLOC 1',
'#undef HAVE_DECL_ALIGNED_ALLOC')
.gsub('#define HAVE_ALLOCA 1',
'#undef HAVE_ALLOCA')
.gsub('#define HAVE_ALLOCA_H 1',
'#undef HAVE_ALLOCA_H')
File.open(filename, 'w') { |f| f.write(content) }
end
def get_ref_info(ref = 'master')
response = `curl "#{LATEST_URL % ref}" 2>/dev/null`
meta = JSON.parse(response).first
{
'sha' => meta['sha'],
'date' => Date.parse(meta['commit']['committer']['date'])
}
end
def apply_patch(patch, target)
raise "ERROR: \"#{target}\" does not exist." unless File.exist?(target)
if patch[:url]
system "mkdir -p \"#{target}/patches\""
patch_file = "#{target}/patches/patch-{num}.diff"
num = 1
num += 1 while File.exist?(patch_file.gsub('{num}', num.to_s.rjust(3, '0')))
patch_file = patch_file.gsub('{num}', num.to_s.rjust(3, '0'))
puts "Downloading patch: #{patch[:url]}"
system "curl -L# \"#{patch[:url]}\" -o \"#{patch_file}\""
puts 'Applying patch...'
system "cd \"#{target}\" && patch -f -p1 -i \"#{patch_file}\""
elsif patch[:replace]
raise 'ERROR: Patch replace input error' unless patch[:replace].size == 3
file, before, after = patch[:replace]
filepath = File.join(target, file)
unless File.exist?(filepath)
raise "ERROR: \"#{file}\" does not exist in #{target}"
end
f = File.open(filepath, 'rb')
s = f.read
sub = s.gsub!(before, after)
raise "ERROR: Replacement filed in #{file}" if sub.nil?
f.reopen(filepath, 'wb').write(s)
f.close
end
end
#
# Run it!
#
main

View File

@@ -1,244 +0,0 @@
#!/usr/bin/env ruby
require 'json'
require 'date'
require 'optparse'
#
# Config
#
DOWNLOAD_URL = "https://github.com/mirrors/emacs/tarball/%s"
LATEST_URL = "https://api.github.com/repos/emacs-mirror/emacs/commits?sha=%s"
ROOT_DIR = File.expand_path('..', __FILE__)
TARBALL_DIR = "#{ROOT_DIR}/tarballs"
SOURCES_DIR = "#{ROOT_DIR}/sources"
BUILDS_DIR = "#{ROOT_DIR}/builds"
#
# Main
#
def main
opts = parse_options
ref = ARGV.shift
meta = get_ref_info(ref)
if meta['sha'] && meta['date']
tarball = download_tarball(meta['sha'])
source = extract_tarball(tarball, patches(opts))
app = compile_source(source)
archive_app(app, meta['sha'], meta['date'])
else
raise "\nERROR: Failed to get commit info from GitHub API."
end
end
#
# Patches
#
def patches(opts = {})
p = []
if opts[:srgb]
p << {
:replace => [ # sRGB patch for older version of nsterm.m.
"src/nsterm.m",
"*col = [NSColor colorWithCalibratedRed: r green: g blue: b alpha: 1.0];",
"*col = [NSColor colorWithDeviceRed: r green: g blue: b alpha: 1.0];"
]
}
end
if opts[:srgb_244]
p << {
:replace => [
"src/nsterm.m",
"return [NSColor colorWithCalibratedRed: red",
"return [NSColor colorWithDeviceRed: red"
]
}
end
p
end
#
# Options
#
def parse_options
options = {:srgb => false, :srgb_244 => false}
OptionParser.new do |opts|
opts.banner = "Usage: ./build-emacs-for-osx [options] [branch/tag/sha]\n" +
"\n" +
"Branch, tag, and SHA are from the mirrors/emacs Github repo,\n" +
"available here: https://github.com/mirrors/emacs\n" +
"\n"
opts.on('--srgb', "Use sRGB patch (pre Emacs 24.4).") do
options[:srgb] = true
end
opts.on('--srgb-244', "Use sRGB patch for 24.3 -> 24.4 dev builds. sRGB patch not needed in final 24.4.") do
options[:srgb] = true
end
end.parse!
options
end
#
# Core Methods
#
def download_tarball(sha)
mkdir TARBALL_DIR
url = (DOWNLOAD_URL % sha)
filename = "emacs-mirror-emacs-#{sha[0..6]}.tgz"
target = File.join(TARBALL_DIR, filename)
if !File.exist?(target)
puts "\nDownloading tarball from GitHub. This could take a while, " +
"please be patient."
system "curl -L \"#{url}\" -o \"#{target}\""
raise "\nERROR: Download failed." unless File.exist?(target)
else
puts "\nINFO: #{filename} already exists locally, attempting to use."
end
target
end
def extract_tarball(filename, patches = [])
mkdir SOURCES_DIR
dirname = File.basename(filename).gsub(/\.\w+$/, '')
target = "#{SOURCES_DIR}/#{dirname}"
if !File.exist?(target)
puts "\nExtracting tarball..."
system "tar -xzf \"#{filename}\" -C \"#{SOURCES_DIR}\""
raise "\nERROR: Tarball extraction failed." unless File.exist?(target)
patches.each do |patch|
apply_patch(patch, target)
end
else
puts "\nINFO: #{dirname} source tree exists, attempting to use."
end
target
end
def compile_source(source)
target = "#{source}/nextstep"
if !File.exist?("#{target}/Emacs.app")
puts "\nCompiling from source. This will take a while..."
if File.exist? "#{source}/autogen/copy_autogen"
system "cd \"#{source}\" && autogen/copy_autogen"
elsif File.exist? "#{source}/autogen.sh"
system "cd \"#{source}\" && ./autogen.sh"
end
system "cd \"#{source}\" && ./configure --with-ns"
system "cd \"#{source}\" && make"
system "cd \"#{source}\" && make install"
raise "\nERROR: Build failed." unless File.exist?("#{target}/Emacs.app")
else
puts "\nINFO: Emacs.app already exists in " +
"\"#{target.gsub(ROOT_DIR + '/', '')}\", attempting to use."
end
"#{target}/Emacs.app"
end
def archive_app(app, sha, date)
mkdir BUILDS_DIR
filename = "Emacs.app-#{date}-(#{sha[0..6]}).tbz"
target = "#{BUILDS_DIR}/#{filename}"
app_base = File.basename(app)
app_dir = File.dirname(app)
if !File.exist?(target)
puts "\nCreating #{filename} archive in \"#{BUILDS_DIR}\"..."
system "cd \"#{app_dir}\" && tar -cjf \"#{target}\" \"#{app_base}\""
else
puts "\nINFO: #{filename} archive exists in " +
"#{BUILDS_DIR.gsub(ROOT_DIR + '/', '')}, skipping archving."
end
end
#
# Helper Methods
#
def mkdir(dir)
system "mkdir -p \"#{dir}\""
end
def get_ref_info(ref = 'master')
response = `curl "#{LATEST_URL % ref}" 2>/dev/null`
meta = JSON.parse(response).first
return {
'sha' => meta['sha'],
'date' => Date.parse(meta['commit']['committer']['date'])
}
end
def apply_patch(patch, target)
raise "ERROR: \"#{target}\" does not exist." unless File.exist?(target)
if patch[:url]
system "mkdir -p \"#{target}/patches\""
patch_file = "#{target}/patches/patch-{num}.diff"
num = 1
while File.exist? patch_file.gsub('{num}', num.to_s.rjust(3, '0'))
num += 1
end
patch_file.gsub!('{num}', num.to_s.rjust(3, '0'))
puts "Downloading patch: #{url}"
system "curl -L# \"#{url}\" -o \"#{patch_file}\""
puts "Applying patch..."
system "cd \"#{target}\" && patch -f -p1 -i \"#{patch_file}\""
elsif patch[:replace]
raise "ERROR: Patch replace input error" unless patch[:replace].size == 3
file, before, after = patch[:replace]
filepath = File.join(target, file)
if !File.exist?(filepath)
raise "ERROR: \"#{file}\" does not exist in #{target}"
end
f = File.open(filepath, 'rb')
s = f.read
sub = s.gsub!(before, after)
raise "ERROR: Replacement filed in #{file}" if sub.nil?
f.reopen(filepath, 'wb').write(s)
f.close
end
end
#
# Run it!
#
main