The Maven plugins that will expose your Git passwords and how Docker helps prove it

A few days ago, I noticed something very troubling after using the Maven release plugin to publish a release artifact to an internal Maven repository: my git password was exposed in the Maven build output as well as a git.properties file that the Maven Git commit ID plugin generated. These files are is now sitting in Artifactory for all to read.

screenshot-example

Not cool. For Maven-based projects, I typically use the the Maven Release Plugin. Because we’d also like to track some of the git metadata about how the build was produced, we also use the Maven Git commit ID plugin as well, which plays quite nicely SpringBoot. So I was very disturbed to see my password all over the place.

What is happening here?

First of all, I should be clear that the project in question is using the Maven command line wrapper and pulling down Maven 3.3.9, which is the latest at the time of this writing. I’m also using Git 2.7.4 and the current release of the Maven Git commit ID plugin, which is was 2.2.0. For the most part, everything is current.

This issue here is not specific plugin any single plugin (but it looks like the Maven Release Plugin is the core offender), but rather the issues only manifest themselves in certain conditions when the group of plugins interact with one another during the release process. The Maven plugins in question are:

The combination of these plugins will expose your Git passwords when using Git over either HTTP or HTTPS when the Maven Release plugins release:prepare and release:perform plugins are invoked, but curiously not when the package,install, or deploy goals are invoked. Additionally if you’re using the Maven Git Commit ID Plugin to capture commit information in your build, the generated git.properties will contain your user name and password when using the default setting and this file will be visible in the Maven repository your artifact is published to. What appears to be happening is that the Maven release plugin in rewriting the git origin URI and including the credentials in the URI. Thus, when the git commit ID plugin goes to resolve the git.remote.origin.url value, it now includes the username and password as well.

Demonstrating the issue

Reproducing this issue was kind of a pain in the ass, but with tools like Docker and Docker Compose, it’s a little less painless. I have created such an environment which does the following:

  • Creates an instance of Artifactory OSS for our Maven repository
  • Creates an instance of Gitbucket for the git repo manager
  • Creates a user named John Yaya who has a username of jyaya and a password of password
  • Creates a container with Maven and Git which also mounts a sample Maven project to demo the issue.

The project is here:

https://github.com/damnhandy/maven-publish-issue

Check the README.md for details on how to run it, but it works fine under Docker Machine on OS X and Linux. This project will startup both Artifactory and Gitbucket and startup a “workspace” container and dump you into the container where you’ll be able to execute the Maven and git commands. The test is 100% repeatable whenever you perform a release.

How you can prevent this?

There are a few ways you can prevent your passwords from being exposed:

Use SSH instead of HTTP/HTTPS in your CI setups

SSH is doesn’t have this issue and won’t expose your passwords. SSH avoids this problem all together, unless of course you use SSH with usernames and passwords. Granted, SSH may not be appropriate in all cases. If you’re in an enterprisey environment, this may be more complicated. SSH is also kind of a pain in the ass on Windows. If SSH isn’t an option, there’s a few more options.

Use Maven Release plugin 2.4.2 or higher

By default, the super pom in Maven 3.3.9 uses version 2.3.2 of the Maven Release plugin. This is fixed as of 2.4.2, but if you don’t define the version to use, you’ll be exposed. You have to explicitly define the version of the Maven Release plugin to use:

<plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-release-plugin</artifactId>
   <version>2.5.3</version>
</plugin>

This will keep your passwords out of yours logs in Jenkins, TravisCI, or other CI environment, but it doesn’t address the fact that the Git commit ID plugin, and probably others, will still render the username and password in the URI.

Exclude the git.remote.origin.url property from your build

If you’re using the Git Commit ID Plugin, exclude the git.remote.origin.url property from your build:

<configuration>
	<excludeProperties>
	  <excludeProperty>git.remote.origin.url</excludeProperty>
	</excludeProperties>
</configuration>

This will completely remove the origin URI from the properties file. This can be annoying if you want to track where the code came from, which handy if you have more than one Git repo manager hosting the same code (i.e. you have a mirror repo that is local to one of your global offices). I have submitted PR #241 which attempts to strip out the password if it’s found in the URI.

Use Git Commit ID Plugin 2.2.1 or higher

Update: as of 3/26/2016, version 2.2.1 of the Git Commit ID Plugin 2.2.1 plugin was released which includes PR #241. And that PR fixes the issue altogether.

Advertisements

Creating containerized build environments with the Jenkins Pipeline plugin and Docker. Well, almost.

Docker and Jenkins are like the chocolate and peanut butter of the DevOps world. The combination of the two present a ton of new opportunities and headaches. I’m going to talk about both.

For this post, I’m assuming you are already familiar with setting up Jenkins and comfortable with Docker. Rather than rehash a lot of existing posts on Jenkins and Docker, I would suggest heading on over to the Riot Games Engineering blog where they have a ton of excellent articles on integrating Docker and Jenkins. I’m going to focus on my specific set up, but I’ve borrowed a lot of ideas from them.

Target setup

I say “target” because all of the pieces to don’t yet do what I’d like them to do. It’s simple really: set up a Jenkins master in a container on one host with multiple JNLP agent containers across multiple hosts. The agent hosts could run in different AWS VPCs and/or accounts using ECS.

My goal here was to have a generic agent configuration that could be deployed onto any host. Each project would then be responsible for for defining its own build environment and that is expressed through a container. This would put the build environment configuration in hands of the development team rather than the team that is managing the Jenkins infrastructure. I REALLY wanted to avoid having agents with a specific set of build tools. Containerized build environments can do this, it’s just getting everything to play nice that is the real challenge.

To get me there, I’m also leveraging the Jenkins Pipeline/Workflow plugin. This set of plugins gives you a very elegant DSL for describing build pipelines. Even better, it has pretty slick support for using containerized build environments via the Cloudbees Docker Pipeline plugin. It’s pretty simple to do something like so:

node('test-agent') {
    stage "Container Prep"
    // do the thing in the container
    docker.image('maven:3.3.3-jdk-8').inside {
        // get the codez
        stage 'Checkout'
        git url: 'https://github.com/damnhandy/Handy-URI-Templates.git'
        stage 'Build'
        // Do the build
        sh "./mvnw clean install"
    }
}

This pipeline will execute the build on a Jenkins agent named “test-agent” and will attempt run the build inside a container based on the “maven”3.3.3-jdk-8” image. This particular pipeline runs fine when the agent runs directly on the host, but it fails when the Jenkins agent runs in a container.

Don’t do Docker in Docker

By having either the Jenkins master or slave in a container, one might assume that I’d need to run the container in privileged mode and doing the whole “Docker-in-Docker” thing. I’m not. Jérôme Petazzoni published very informative post titled “Using Docker-in-Docker for your CI or testing environment? Think twice.” You should read it. The rest of this post assumes that you did.

If you’re still using a copy of the wrapdocker script, you should ask yourself “why?”. It’s much simpler to do something like so:

docker run -v ${JENKINS_HOME}:/var/jenkins_home \
     -v /var/run/docker.sock:/var/run/docker.sock \
     -v $(which docker):/bin/docker -p 8080:8080 \
     -p 50000:50000 damnhandy/jenkins

This will bring up Jenkins and it will be able to call the docker command and do everything a “Docker-in-Docker” set can do. There’s no need for privileged mode or the wrapdocker script.

One caveat here: you’re not going to be able to simply reuse the official Jenkins image to do this because the jenkins user needs to be a part of the docker and/or users group in order to be able to make use of the socket. Once you do that, Jenkins can happily call docker from within the container, and you can build and run other containers with ease.

The Jenkins JNLP agent container

The Jenkins agent container follows similar rules as the master. It too needs access to the docker socket and executable and you can do something like this:

docker run -v ${JENKINS_HOME}:/var/jenkins_home \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $(which docker):/bin/docker --name=jenkins-slave \
    -d damnhandy/jenkins-slave -url http://192.168.99.100:8080/ \
    a0a1b92971030d5f5dd69bd972c6cd899f705ddd3699ca3c5e92f937d860be7e test-agent

Like the Jenkins master, you have to ensure that the jenkins user is in a group that has the privileges to access the docker socket. I’m using a fork of the Jenkins JNLP slave container and adding the necessary groups. Once you do this, your agent will come up and you’ll be able to execute builds against the agent. Almost.

The exact moment where the wheels came off

The moment you start to execute a build that runs with in a container, things go off the rails pretty quickly. The problem is that you have the agent container binding to a host directory ${JENKINS_HOME}:/var/jenkins_home and then the build container needs access to the same directory. The Cloudbees Docker Pipeline plugin will execute the following when using the docker.inside() function:

docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/uri-templates-in-docker \
-v /var/jenkins_home/workspace/uri-templates-in-docker:/var/jenkins_home/workspace/uri-templates-in-docker:rw \
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** \
-e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** \
maven:3.3.3-jdk-8 cat

The container is trying to mount the the host directory /var/jenkins_home/workspace/uri-templates-in-docker into this containerized build environment for Maven 3.3.3 and tries to set that directory as the current working directory. This all works great if the Jenkins agent is running directly on the host, outside of a container. When running inside a container, I’m basically trying to do this:

And this absolutely does not work. Because I’m mapping the docker socket from the host to the Jenkins agent container,  any volumes that are mounted in this “faux docker-in-docker” manner are actually referenced from the host, not from perspective of the Jenkins agent container. So assuming the directory of the ${JENKINS_HOME} on the host was something like /opt/jenkins_home, something like this “should” work:

docker run -t -d -u 1000:1000 -w /opt/jenkins_home/workspace/uri-templates-in-docker \
-v /opt/jenkins_home:/var/jenkins_home/workspace/uri-templates-in-docker:rw \
-e ******** 
maven:3.3.3-jdk-8 cat

But there’s a few problems with this approach:

  • Since we’re kind of running “docker-in-docker”, getting the path of host directory is tricky.
  • It’s not exactly portable since the containers need to have more intimate knowledge of the hosts directory structure.

There is a better way.

The beauty of Docker data volume containers

It’s taken me about 18 months to finally understand why one would want to use a container for storing data. Now I get it. For this use case, a docker volume container is an incredibly elegant way of sharing a volume between multiple containers. It provides a clean abstraction around the volume and provides a host-independent way of referencing the volume. With data volume containers, you end up with something like this:

Again, borrowing some ideas from Maxfield at Riot Games, I created a data volume container pretty much the same way he describes. Now while Docker 1.9+ have the ability to create named volumes, theres a few major issues with using them right now:

  • The documentation is seriously lacking. And when I say lacking, I mean it doesn’t exist. See issue #20465
  • Volumes created with docker volume create will always be owned by root. This is being fixed for Docker 1.11, but it doesn’t help much when you’re using docker 1.9 and 1.10. Since Jenkins runs as jenkins, this doesn’t work.

Since my target environment is Amazon ECS which is using Docker 1.9, I’ll continue with data volume containers. I used Maxfield’s Dockerfile verbatim and created the container like so:

docker create --name=jenkins-data damnhandy/jenkins-data

And now start the Jenkins agent like so:

docker run --volumes-from=jenkins-data \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $(which docker):/bin/docker --name=jenkins-slave \
    -d damnhandy/jenkins-slave -url http://192.168.99.100:8080/ \
    a0a1b92971030d5f5dd69bd972c6cd899f705ddd3699ca3c5e92f937d860be7e test-agent

So far so good. The bad news is that the Docker Pipeline plugin still insists on mounting a volume from the host, which in my case doesn’t actually exist in the Jenkins agent container. So for now, the Cloudbees Docker Pipeline plugin is a non-starter.

However, it is possible to bypass the Docker Pipeline plugin and change the pipeline script to be as follows:

node('test-agent') {
    // Get some code from a GitHub repository
    git url: 'https://github.com/damnhandy/Handy-URI-Templates.git'
    sh 'docker run -t -u 1000:1000 --volumes-from=jenkins-data -w /var/jenkins_home/workspace/uri-templates-in-docker maven:3.3.3-jdk-8 ./mvnw package'
}

And this mostly works. The project will build but fails on the tests because the pipeline git task doesn’t handle submodules very well. However, this is an issue with the specific project and we at least have the build failing 2/3’s the way through the build in the target containerized build enviornment.

Wrapping Up

Containerized build environments are such a great idea and will save a lot of hassle down the line. I’m also loving the new Jenkins pipeline plugins, even though there’s a few rough edges.  I’ve posted a the code for a working environment that illustrates this set up here:

https://github.com/damnhandy/jenkins-pipeline-docker

Remember, not everything here works as desired, but it at demos what could be possible. I hope this post helped folks better understand how to execute docker builds in a Jenkins container and get better grasp of how docker manages data volumes. There’s still a lot more to learn and few PRs to create 😉

I can’t make a Windows 10 Mobile app on a Mac, and that’s a problem

So now that Apple has shown it’s hand with the iPhone 6s and Google set to unveil new Nexus devices at the end of the month, one may (or may not) start wondering about what’s next for Microsoft, which will be holding their event in October. While Microsoft may release some interesting hardware at the October event, hardware isn’t the major problem for Windows 10 Mobile, it’s the ecosystem. It sucks. The apps that you can get for iOS and Android simply aren’t there for Windows Mobile. I’d love to give Windows 10 Mobile a shot, but I really need access to the MBTA mTicket app, which hasn’t been available for anything other than iOS and Android. Microsoft is desperately trying to woo developers to its mobile platform, but they seem to be missing the mark with their recent moves.

Most Mobile Developers use Macs

If you’re going to develop an iOS App, you need a Mac. Even if you choose to develop on another OS, a Mac comes into the picture eventually, either in the form of a server of hosted service. If you’re already developing and supporting iOS apps, you’re probably using Mac OS X already as your primary environment. If you’re interested in making an Android version of your app, it’s pretty painless to snag Android Studio and SDKs and run them along side XCode. Easy. It also doesn’t hurt that iOS and Android come from familiar UNIX-y underpinnings. You likely don’t even need a Windows PC because you already have everything you need. What you do need are emulators and actual devices – not Windows.

React Native is Mac-only

React Native is one of those tools that I think underscores the situation. React Native is getting a lot of attention lately, but it’s actual real-world usage is still relatively small. I bring up React Native as it kind supports where the majority of mobile development is taking place. At present React Native for both iOS and Android, is currently only available for Mac OS X. While it is possible for React Native for Android to run on Windows and Linux, it’s currently not supported. At present, React Native has the following dependencies:

Only one of these dependencies is cross-platform. While yes, Homebrew could easily be swapped out for apt-get or OneGet in Windows 10, learning the details of how to distribute this stuff non-OS X platforms is a sizable time investment.

VisualStudio 2015 for Windows 10 Mobile Development won’t help

Microsoft is investing into tooling that caters to existing Windows developers but does nothing to appeal to existing mobile developers on non-Windows platforms. In fact, adding VisualStudio 2015 to an existing workflow presents more work to existing mobile developers. They now need to consider adding Windows infrastructure to support what is currently a niche mobile platform. Adding Windows 10 Mobile into an existing Mac or Linux-based development is a pretty big commitment. If you’re already invested in a Mac OS X development environment, you now need a Windows PC or a Virtual Machine running Windows. You also need a copy of Visual Studio. If you’re running a business, Visual Studio Community isn’t an option and you need to shell out $1,200 per user. A VM is a lousy option given the capacity demands of Visual Studio (30+GB) and that the Windows Mobile emulator doesn’t play nice when running in a VM.

The release of VisualStudio 2015 is a pretty big deal. It added a lot of nice new capabilities for mobile developers. The Windows Bridges are a great idea and might help in the long term. While it’s cool that you can get up and running quickly with existing Objective-C and Android Java code, it glosses over the fact that existing mobile devs coming from a Mac/UNIX background, probably don’t know the Windows/VisualStudio way of doing things. Coming from a non-Windows development background, learning the VisualStudio developer tool chain is awkward. In fact, it sucks because it feels so weird. That’s a learning curve that those comfortable with UNIX-based build tools aren’t going to enjoy much. It also takes time. If you’re a small shop, you are going to seriously question if it’s worth going through these shenanigans for roughly 2.7% of the mobile market.

Windows 10 Mobile is a minor player that thinks it’s in the majors

While Microsoft seems to acknowledge that Windows 10 Mobile is a minority platform, the developer tools side of the house doesn’t seem to get this yet. The browser team seems to understand that a lack of testing for IE was due to the fact that a lot of design shops were ignoring IE. Either they didn’t have access to Windows PCs running IE or they were ignoring it due to cost issues. This ultimately led to Modern.ie, which provides Windows VMs with various versions of Edge and IE. This is an incredibly helpful resource for all web developers.  Mobile developers have no such resources available for Windows 10 Mobile. Given Windows Mobile’s relatively insignificant market share, it’s easier to just ignore the platform outright.

How about a Window 10 Mobile Tools that run on Mac OS X and Linux?

VisualStudio Code is a nice light-weight developer tool. It’s great for working with Node.js and the early builds of ASP.NET 5, but it could be so much more. It would be SUPER great if this were the tool that could help developers get started with the Windows Mobile on non-Windows platforms. Toss in a Windows 10 Mobile emulator that runs on non-Windows OS’s and now you’re cooking with gas. To be able to debug and test something like a Cordova-based app through a Window Mobile emulator on Mac OS X would be a huge leap forward. Hell, you might even see a version of React Native for Windows 10 Mobile! Top it off with being able to publish to the Windows Store from a bash shell, and you’re doing that much, much better. And these tools also need to be free – just like XCode and Android Studio.

App developers aren’t going to flee from Mac OS X to run Windows 10 and VisualStudio 2015 because of the new tooling in VisualStudio 2015. XCode users weren’t really pining for Obective-C support in VisualStudio as that’s not their jam. But add C# and Windows Mobile tooling as XCode plugins or through VS Code, things get a lot more interesting. That’s not so crazy now given that the .NET 5 beta runs on Mac OS X and should be final by 2016.

Microsoft desperately needs to put out tools for Windows Mobile developers on the platforms they are currently developing on – not just Windows. If Microsoft keeps pretending that the majority of mobile developers want VisualStudio, they’ll still be struggling to crack that 3% market share. These folks don’t do Windows.

Handy URI Templates 2.0.3 Released

I have put out a new release of Handy URI Templates this morning that fixes a few issues and adds some new features such as the ability to perform partial template expansion. A big thanks to Christoph Nagel for the pull request! Please file an issue if you have any problems with this release. Up next, I plan on fixing a few issues that some Android developers have been hitting and then eventually get around to reverse matching.