Switch to using QEMU VMs for building calibre on macOS and Windows

The VM performance is better and QEMU is more
hackable/automatable than VirtualBox. Also paves the way to do the
building on remote servers if needed.

bypy/README.rst

@@ -20,8 +20,10 @@ First create some empty top level directory and run the following commands::
     cd calibre
     ./setup.py bootstrap
 
-To make the Windows and macOS builds it uses VirtualBox VMs. Instructions on
-creating the VMs are in their respective sections below.
+To make the Windows and macOS builds it uses QEMU VMs. Instructions on
+creating the VMs are in the bypy repo under :file:`virtual_machine/README.rst`.
+Required software for the VMs are listed in :file:`bypy/windows.conf` and
+:file:`bypy/macos.conf`.
 
 Linux
 -------
@@ -43,16 +45,8 @@ The output will be in :literal:`dist`
 macOS
 --------------
 
-You need a VirtualBox virtual machine of macOS 10.14 (Mojave). Name the
-VM using ``vm_name`` from :literal:`bypy/macos.conf`. To setup macOS inside the VM,
-follow the steps:
-
-* Turn on Remote Login under Network (SSHD)
-* Create a user account named ``kovid`` and enable password-less login for SSH
-  for that account (setup ``~/.ssh/authorized_keys``)
-* Setup ssh into the VM from the host using the ``vm_name`` from above.
-* Install the needed software mentioned in :literal:`bypy/macos.conf`.
-
+Name the QEMU VM using ``vm_name`` from :literal:`bypy/macos.conf`.
+Make sure all software mentioned in :file:`bypy/macos.conf` is installed.
 To build the dependencies for calibre, run::
 
     ./setup.py build_dep macos
@@ -68,58 +62,8 @@ The output will be in :literal:`dist`
 Windows
 -------------
 
-You need a VirtualBox virtual machine of Windows 7 64bit. Name the
-VM using ``vm_name`` from :literal:`bypy/windows.conf`. To setup windows inside the VM,
-follow the steps:
-
-* Install all the software mentioned in :literal:`bypy/windows.conf`
-
-* Install cygwin, with the: vim, dos2unix, rsync, openssh, unzip, wget, make, zsh, patch, bash-completion, curl
-  packages
-
-* Edit ``/etc/passwd`` and replace all occurrences of ``/bin/bash`` with ``/bin/zsh`` (in
-  a cygwin prompt)
-
-* Setup a password for your windows user account
-
-* Follow the steps here: http://pcsupport.about.com/od/windows7/ht/auto-logon-windows-7.htm to allow the
-  machine to bootup without having to enter the password
-
-* The following steps must all be run in an administrator cygwin shell, to
-  enable SSH logins to the machine
-
-* First clean out any existing cygwin ssh setup with::
-
-    net stop sshd
-    cygrunsrv -R sshd
-    net user sshd /DELETE
-    net user cyg_server /DELETE (delete any other cygwin users account you
-    can list them with net user)
-    rm -R /etc/ssh*
-    mkpasswd -cl > /etc/passwd
-    mkgroup --local > /etc/group
-
-* Assign the necessary rights to the normal user account (administrator
-  cygwin command prompt needed - editrights is available in ``\cygwin\bin``)::
-
-    editrights.exe -a SeAssignPrimaryTokenPrivilege -u kovid
-    editrights.exe -a SeCreateTokenPrivilege -u kovid
-    editrights.exe -a SeTcbPrivilege -u kovid
-    editrights.exe -a SeServiceLogonRight -u kovid
-
-* Run::
-
-    ssh-host-config
-    And answer (yes) to all questions. If it asks do you want to use a
-    different user name, specify the name of your user account and enter
-    username and password
-
-* Start sshd with::
-
-    net start sshd
-
-* See http://www.kgx.net.nz/2010/03/cygwin-sshd-and-windows-7/ for details
-
+Name the QEMU VM using ``vm_name`` from :file:`bypy/windows.conf`.
+Make sure all software mentioned in :file:`bypy/windows.conf` is installed.
 
 To build the dependencies for calibre, run::
 

bypy/macos.conf

@@ -1,6 +1,6 @@
-# Requires installation of XCode 10.2 and Python 3.7 and 
+# Requires installation of XCode 10.3 and Python 3 and
 # python3 -m pip install certifi
 
-vm_name 'calibre-macos-build'
+vm_name 'macos-calibre-build'
 root '/Users/Shared/calibre-build'
-python '/Library/Frameworks/Python.framework/Versions/3.7/bin/python3'
+python '/usr/local/bin/python3'

bypy/macos/sign.py

@@ -18,13 +18,13 @@
 
 from bypy.utils import current_dir, run_shell, timeit
 
-CODESIGN_CREDS = os.path.expanduser('~/cert-cred')
-CODESIGN_CERT = os.path.expanduser('~/maccert.p12')
+CODESIGN_CREDS = os.path.expanduser('~/code-signing/cert-cred')
+CODESIGN_CERT = os.path.expanduser('~/code-signing/maccert.p12')
 # The apple id file contains the apple id and an app specific password which
 # can be generated from appleid.apple.com
 # Note that apple accounts require two-factor authentication which is currntly
 # setup on ox and via SMS on my phone
-APPLE_ID = os.path.expanduser('~/aid')
+APPLE_ID = os.path.expanduser('~/code-signing/apple-notarization-creds')
 path_to_entitlements = os.path.expanduser('~/calibre-entitlements.plist')
 
 

bypy/windows.conf

@@ -1,4 +1,4 @@
-# Requires installation of Visual Studio 2017 Community Edition, WiX Toolset, Git, Ruby, Python 3.7 and Perl
+# Requires installation of Visual Studio 2017 Community Edition, WiX Toolset, Git, Ruby, Python and Perl
 # git.exe must be in PATH. Must have ~120GB available disk space and 8GB RAM
 # Install certifi in python 3 with:
 # py.exe -m pip install certifi
@@ -8,9 +8,9 @@
 # Ensure the windwos sdk version at least 10.0.18362 or newer is installed, needed for
 # qt-webengine-14.1
 
-vm_name 'calibre-windows-build'
+vm_name 'windows-calibre-build'
 root 'C:/r'
 python 'py.exe'
 perl 'C:/Strawberry/perl/bin/perl.exe'
-ruby 'C:/Ruby26-x64/bin/ruby.exe'
+ruby 'C:/Ruby27-x64/bin/ruby.exe'
 mesa 'C:/mesa'

bypy/windows/__main__.py

@@ -444,17 +444,20 @@ def build_portable(env):
 
 
 def sign_files(env, files):
+    with open(os.path.expandvars(r'${HOMEDRIVE}${HOMEPATH}\code-signing\cert-cred')) as f:
+        pw = f.read().strip()
+    CODESIGN_CERT = os.path.abspath(os.path.expandvars(r'${HOMEDRIVE}${HOMEPATH}\code-signing\authenticode.pfx'))
     args = [SIGNTOOL, 'sign', '/a', '/fd', 'sha256', '/td', 'sha256', '/d',
             'calibre - E-book management', '/du',
-            'https://calibre-ebook.com', '/tr']
+            'https://calibre-ebook.com', '/f', CODESIGN_CERT, '/p', pw, '/tr']
 
     def runcmd(cmd):
         for timeserver in ('http://sha256timestamp.ws.symantec.com/sha256/timestamp', 'http://timestamp.comodoca.com/rfc3161',):
             try:
                 subprocess.check_call(cmd + [timeserver] + list(files))
                 break
             except subprocess.CalledProcessError:
-                print ('Signing failed, retrying with different timestamp server')
+                print('Signing failed, retrying with different timestamp server')
         else:
             raise SystemExit('Signing failed')