diff options
| -rw-r--r-- | protoc-artifacts/README.md | 222 | ||||
| -rwxr-xr-x | protoc-artifacts/build-protoc.sh | 55 | ||||
| -rw-r--r-- | protoc-artifacts/pom.xml | 63 |
3 files changed, 167 insertions, 173 deletions
diff --git a/protoc-artifacts/README.md b/protoc-artifacts/README.md index 5c5ca4ce..9b148891 100644 --- a/protoc-artifacts/README.md +++ b/protoc-artifacts/README.md @@ -7,13 +7,31 @@ build and publish a ``protoc`` executable (a.k.a. artifact) to Maven repositories. The artifact can be used by build automation tools so that users would not need to compile and install ``protoc`` for their systems. +If you would like us to publish protoc artifact for a new platform, please send +us a pull request to add support for the new platform. You would need to change +the following files: + +* [build-protoc.sh](build-protoc.sh): script to cross-build the protoc for your + platform. +* [pom.xml](pom.xml): script to upload artifacts to maven. +* [build-zip.sh](build-zip.sh): script to package published maven artifacts in + our release page. + +## Maven Location +The published protoc artifacts are available on Maven here: + + http://central.maven.org/maven2/com/google/protobuf/protoc/ + ## Versioning The version of the ``protoc`` artifact must be the same as the version of the Protobuf project. ## Artifact name The name of a published ``protoc`` artifact is in the following format: -``protoc-<version>-<os>-<arch>.exe``, e.g., ``protoc-3.0.0-alpha-3-windows-x86_64.exe``. +``protoc-<version>-<os>-<arch>.exe``, e.g., ``protoc-3.6.1-linux-x86_64.exe``. + +Note that artifacts for linux/macos also have the `.exe` suffix but they are +not windows binaries. ## System requirement Install [Apache Maven](http://maven.apache.org/) if you don't have it. @@ -29,98 +47,116 @@ generate the configure script. Under the protobuf project directory: + +``` +$ ./autogen.sh +``` + +### Build the artifact for each platform + +Run the build-protoc.sh script under this protoc-artifacts directory to build the protoc +artifact for each platform. For example: + ``` -$ ./autogen.sh && ./configure && make +$ cd protoc-artifacts +$ ./build-protoc.sh linux x86_64 protoc ``` -## To install artifacts locally -The following command will install the ``protoc`` artifact to your local Maven repository. +The above command will produce a `target/linux/x86_64/protoc` binary under the +protoc-artifacts directory. + +For a list of supported platforms, see the comments in the build-protoc.sh +script. We only use this script to build artifacts on Ubuntu and MacOS (both +with x86_64, and do cross-compilation for other platforms. + +### Tips for building for Linux +We build on Centos 6.9 to provide a good compatibility for not very new +systems. We have provided a ``Dockerfile`` under this directory to build the +environment. It has been tested with Docker 1.6.1. + +To build a image: + ``` -$ mvn install +$ docker build -t protoc-artifacts . ``` -## Cross-compilation -The Maven script will try to detect the OS and the architecture from Java -system properties. It's possible to build a protoc binary for an architecture -that is different from what Java has detected, as long as you have the proper -compilers installed. +To run the image: + +``` +$ docker run -it --rm=true protoc-artifacts bash +``` -You can override the Maven properties ``os.detected.name`` and -``os.detected.arch`` to force the script to generate binaries for a specific OS -and/or architecture. Valid values are defined as the return values of -``normalizeOs()`` and ``normalizeArch()`` of ``Detector`` from -[os-maven-plugin](https://github.com/trustin/os-maven-plugin/blob/master/src/main/java/kr/motd/maven/os/Detector.java). -Frequently used values are: -- ``os.detected.name``: ``linux``, ``osx``, ``windows``. -- ``os.detected.arch``: ``x86_32``, ``x86_64``, ``ppcle_64`` +To checkout protobuf (run within the container): -For example, MinGW32 only ships with 32-bit compilers, but you can still build -32-bit protoc under 64-bit Windows, with the following command: ``` -$ mvn install -Dos.detected.arch=x86_32 +$ # Replace v3.5.1 with the version you want +$ wget -O - https://github.com/google/protobuf/archive/v3.5.1.tar.gz | tar xvzp ``` +### Windows build +We no longer use scripts in this directory to build windows artifacts. Instead, +we use Visual Studio 2015 to build our windows release artifacts. See our +[kokoro windows build scripts here](../kokoro/release/protoc/windows/build.bat). + +To upload windows artifacts, copy the built binaries into this directory and +put it into the target/windows/(x86_64|x86_32) directory the same way as the +artifacts for other platforms. That will allow the maven script to find and +upload the artifacts to maven. + ## To push artifacts to Maven Central Before you can upload artifacts to Maven Central repository, make sure you have read [this page](http://central.sonatype.org/pages/apache-maven.html) on how to configure GPG and Sonatype account. -You need to perform the deployment for every platform that you want to -support. DO NOT close the staging repository until you have done the -deployment for all platforms. Currently the following platforms are supported: -- Linux (x86_32, x86_64, ppcle_64 and cross compiled aarch_64 or ppcle_64) -- Windows (x86_32 and x86_64) with - - Cygwin64 with MinGW compilers (x86_64) - - MSYS with MinGW32 (x86_32) - - Cross compile in Linux with MinGW-w64 (x86_32, x86_64) -- MacOSX (x86_32 and x86_64) - -As for MSYS2/MinGW64 for Windows: protoc will build, but it insists on -adding a dependency of `libwinpthread-1.dll`, which isn't shipped with -Windows. +Before you do the deployment, make sure you have built the protoc artifacts for +every supported platform and put them under the target directory. Example +target directory layout: + + + pom.xml + + target + + linux + + x86_64 + protoc.exe + + x86_32 + protoc.exe + + macos + + x86_64 + protoc.exe + + x86_32 + protoc.exe + + windows + + x86_64 + protoc.exe + + x86_32 + protoc.exe + +You will need to build the artifacts on multiple machines and gather them +together into one place. Use the following command to deploy artifacts for the host platform to a staging repository. + ``` $ mvn clean deploy -P release ``` + It creates a new staging repository. Go to https://oss.sonatype.org/#stagingRepositories and find the repository, usually -in the name like ``comgoogle-123``. - -You will want to run this command on a different platform. Remember, in -subsequent deployments you will need to provide the repository name that you -have found in the first deployment so that all artifacts go to the same -repository: -``` -$ mvn clean deploy -P release -Dstaging.repository=comgoogle-123 -``` - -A 32-bit artifact can be deployed from a 64-bit host with -``-Dos.detected.arch=x86_32`` - -An arm64 artifact can be deployed from x86 host with -``-Dos.detected.arch=aarch_64`` - -A ppcle_64 artifact can be deployed from x86 host with -``-Dos.detected.arch=ppcle_64`` - -A windows artifact can be deployed from a linux machine with -``-Dos.detected.name=windows`` - -When you have done deployment for all platforms, go to -https://oss.sonatype.org/#stagingRepositories, verify that the staging -repository has all the binaries, close and release this repository. +in the name like ``comgoogle-123``. Verify that the staging repository has all +the binaries, close and release this repository. ## Upload zip packages to github release page. After uploading protoc artifacts to Maven Central repository, run the build-zip.sh script to bulid zip packages for these protoc binaries and upload these zip packages to the download section of the github release. For example: + ``` $ ./build-zip.sh 3.6.0 ``` -The above command will create 6 zip files: + +The above command will create 5 zip files: + ``` dist/protoc-3.6.0-win32.zip dist/protoc-3.6.0-osx-x86_32.zip @@ -129,70 +165,14 @@ dist/protoc-3.6.0-linux-x86_32.zip dist/protoc-3.6.0-linux-x86_64.zip dist/protoc-3.6.0-linux-ppcle_64.zip ``` + Before running the script, make sure the artifacts are accessible from: http://repo1.maven.org/maven2/com/google/protobuf/protoc/ -### Tips for deploying on Linux -We build on Centos 6.6 to provide a good compatibility for not very new -systems. We have provided a ``Dockerfile`` under this directory to build the -environment. It has been tested with Docker 1.6.1. - -To build a image: -``` -$ docker build -t protoc-artifacts . -``` - -To run the image: -``` -$ docker run -it --rm=true protoc-artifacts bash -``` - -To checkout protobuf (run within the container): -``` -$ # Replace v3.5.1 with the version you want -$ wget -O - https://github.com/google/protobuf/archive/v3.5.1.tar.gz | tar xvzp -``` - -### Tips for deploying on Windows -Under Windows the following error may occur: ``gpg: cannot open tty `no tty': -No such file or directory``. This can be fixed by configuring gpg through an -active profile in ``.m2\settings.xml`` where also the Sonatype password is -stored: -```xml -<settings> - <servers> - <server> - <id>sonatype-nexus-staging</id> - <username>[username]</username> - <password>[password]</password> - </server> - </servers> - <profiles> - <profile> - <id>gpg</id> - <properties> - <gpg.executable>gpg</gpg.executable> - <gpg.passphrase>[password]</gpg.passphrase> - </properties> - </profile> - </profiles> - <activeProfiles> - <activeProfile>gpg</activeProfile> - </activeProfiles> -</settings> -``` - -### Tested build environments +## Tested build environments We have successfully built artifacts on the following environments: - Linux x86_32 and x86_64: - - Centos 6.6 (within Docker 1.6.1) - - Ubuntu 14.04.2 64-bit -- Linux ppc64le: - - Debian 9.4 - - Cross compiled with `g++-powerpc64le-linux-gnu` on Debian 9.4 x86_64 -- Linux aarch_64: Cross compiled with `g++-aarch64-linux-gnu` on Ubuntu 14.04.2 64-bit -- Windows x86_32: MSYS with ``mingw32-gcc-g++ 4.8.1-4`` on Windows 7 64-bit -- Windows x86_32: Cross compile with ``i686-w64-mingw32-g++ 4.8.2`` on Ubuntu 14.04.2 64-bit -- Windows x86_64: Cygwin64 with ``mingw64-x86_64-gcc-g++ 4.8.3-1`` on Windows 7 64-bit -- Windows x86_64: Cross compile with ``x86_64-w64-mingw32-g++ 4.8.2`` on Ubuntu 14.04.2 64-bit + - Centos 6.9 (within Docker 1.6.1) + - Ubuntu 14.04.5 64-bit +- Linux aarch_64: Cross compiled with `g++-aarch64-linux-gnu` on Ubuntu 14.04.5 64-bit - Mac OS X x86_32 and x86_64: Mac OS X 10.9.5 diff --git a/protoc-artifacts/build-protoc.sh b/protoc-artifacts/build-protoc.sh index c50a1207..076e6b48 100755 --- a/protoc-artifacts/build-protoc.sh +++ b/protoc-artifacts/build-protoc.sh @@ -1,24 +1,34 @@ #!/bin/bash -# Builds protoc executable into target/protoc.exe; optionally build protoc -# plugins into target/protoc-gen-*.exe -# To be run from Maven. -# Usage: build-protoc.sh <OS> <ARCH> <TARGET> -# <OS> and <ARCH> are ${os.detected.name} and ${os.detected.arch} from os-maven-plugin -# <TARGET> can be "protoc" or "protoc-gen-javalite" +# Builds protoc executable into target/<OS>/<ARCH>/protoc.exe; optionally builds +# protoc plugins into target/<OS>/<ARCH>/protoc-gen-*.exe # -# The script now supports cross-compiling windows, linux-arm64, and linux-ppc64le in linux-x86 -# environment. Required packages: -# - Windows: i686-w64-mingw32-gcc (32bit) and x86_64-w64-mingw32-gcc (64bit) -# - Arm64: g++-aarch64-linux-gnu -# - Ppc64le: g++-powerpc64le-linux-gcc +# Usage: ./build-protoc.sh <OS> <ARCH> <TARGET> +# +# <TARGET> can be "protoc" or "protoc-gen-javalite". Supported <OS> <ARCH> +# combinations: +# HOST <OS> <ARCH> <COMMENT> +# cygwin windows x86_32 Requires: i686-w64-mingw32-gcc +# cygwin windows x86_64 Requires: x86_64-w64-mingw32-gcc +# linux linux aarch_64 Requires: g++-aarch64-linux-gnu +# linux linux x86_32 +# linux linux x86_64 +# linux windows x86_32 Requires: i686-w64-mingw32-gcc +# linux windows x86_64 Requires: x86_64-w64-mingw32-gcc +# macos osx x86_32 +# macos osx x86_64 +# mingw windows x86_32 +# mingw windows x86_64 +# +# Before running this script, make sure you have generated the configure script +# in the parent directory (i.e., run ./autogen.sh there). OS=$1 ARCH=$2 MAKE_TARGET=$3 if [[ $# < 3 ]]; then - echo "No arguments provided. This script is intended to be run from Maven." + echo "Not enough arguments provided." exit 1 fi @@ -166,13 +176,8 @@ checkDependencies () echo "Building protoc, OS=$OS ARCH=$ARCH TARGET=$TARGET" -# Nested double quotes are unintuitive, but it works. -cd "$(dirname "$0")" - -WORKING_DIR=$(pwd) CONFIGURE_ARGS="--disable-shared" -TARGET_FILE=target/$MAKE_TARGET.exe if [[ "$OS" == windows ]]; then MAKE_TARGET="${MAKE_TARGET}.exe" fi @@ -253,10 +258,18 @@ fi export CXXFLAGS LDFLAGS -cd "$WORKING_DIR"/.. && ./configure $CONFIGURE_ARGS && - cd src && make clean && make $MAKE_TARGET && - cd "$WORKING_DIR" && mkdir -p target && - cp ../src/$MAKE_TARGET $TARGET_FILE || +# Nested double quotes are unintuitive, but it works. +cd "$(dirname "$0")" + +WORKING_DIR="$(pwd)" +BUILD_DIR="build/$OS/$ARCH" +TARGET_FILE="target/$OS/$ARCH/$MAKE_TARGET.exe" + +mkdir -p "$BUILD_DIR" && cd "$BUILD_DIR" && + ../../../../configure $CONFIGURE_ARGS && + cd src && make $MAKE_TARGET -j8 && + cd "$WORKING_DIR" && mkdir -p $(dirname $TARGET_FILE) && + cp $BUILD_DIR/src/$MAKE_TARGET $TARGET_FILE || exit 1 if [[ "$OS" == osx ]]; then diff --git a/protoc-artifacts/pom.xml b/protoc-artifacts/pom.xml index 667a0215..390cd220 100644 --- a/protoc-artifacts/pom.xml +++ b/protoc-artifacts/pom.xml @@ -33,38 +33,9 @@ </connection> </scm> <build> - <extensions> - <extension> - <groupId>kr.motd.maven</groupId> - <artifactId>os-maven-plugin</artifactId> - <version>1.5.0.Final</version> - </extension> - </extensions> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> - <artifactId>exec-maven-plugin</artifactId> - <version>1.1.1</version> - <executions> - <execution> - <phase>compile</phase> - <goals> - <goal>exec</goal> - </goals> - </execution> - </executions> - <configuration> - <executable>bash</executable> - <arguments> - <argument>build-protoc.sh</argument> - <argument>${os.detected.name}</argument> - <argument>${os.detected.arch}</argument> - <argument>protoc</argument> - </arguments> - </configuration> - </plugin> - <plugin> - <groupId>org.codehaus.mojo</groupId> <artifactId>build-helper-maven-plugin</artifactId> <version>1.8</version> <executions> @@ -77,8 +48,38 @@ <configuration> <artifacts> <artifact> - <file>${basedir}/target/protoc.exe</file> - <classifier>${os.detected.name}-${os.detected.arch}</classifier> + <file>${basedir}/target/linux/x86_64/protoc.exe</file> + <classifier>linux-x86_64</classifier> + <type>exe</type> + </artifact> + <artifact> + <file>${basedir}/target/linux/x86_32/protoc.exe</file> + <classifier>linux-x86_32</classifier> + <type>exe</type> + </artifact> + <artifact> + <file>${basedir}/target/windows/x86_64/protoc.exe</file> + <classifier>windows-x86_64</classifier> + <type>exe</type> + </artifact> + <artifact> + <file>${basedir}/target/windows/x86_32/protoc.exe</file> + <classifier>windows-x86_32</classifier> + <type>exe</type> + </artifact> + <artifact> + <file>${basedir}/target/osx/x86_64/protoc.exe</file> + <classifier>osx-x86_64</classifier> + <type>exe</type> + </artifact> + <artifact> + <file>${basedir}/target/osx/x86_32/protoc.exe</file> + <classifier>osx-x86_32</classifier> + <type>exe</type> + </artifact> + <artifact> + <file>${basedir}/target/linux/aarch_64/protoc.exe</file> + <classifier>linux-aarch_64</classifier> <type>exe</type> </artifact> </artifacts> |