Date: Tue, 13 Jun 2017 11:47:53 -0400
Message-ID: 
Subject: Re: Quick sync re: dep
From: Russ Cox 
To: Andrew Gerrand 
Cc: Peter Bourgon , Edward Muller , Sam Boyer , Jessica Frazelle 

On Sat, Jun 10, 2017 at 3:40 AM, Andrew Gerrand <adg@golang.org> wrote:
> Hi all,
>
> My sincere apologies for missing yesterday's meeting.
> An urgent personal matter arose, and so I completely forgot my schedule.
> (All is fine now.)
>
> Did anyone take notes at the meeting?
>
> Best wishes,
> Andrew

Hi Andrew,

No worries. I took some personal hand-written notes about things for me to
look into in more detail, which I'll reproduce in expanded form below. If
anyone raised anything that I'm missing here, please let me know. Thanks.

My general goal was to ask "Is the big picture reasonable? Are there
objections? Are there things we need to work out among ourselves before
making something like this doc public?"

- Sam pointed out that the big difference between my sketch and dep is that
manifests end up being per-package instead of per-project, which he was
concerned may cause incompatibilities between the two schemes. The big
difference is that packages have no cycles while projects may (and do, in
the case of at least k8s and docker). It's unclear without more
investigation exactly how big a difference this is, and it wasn't one I
intended explicitly. I need to look more into the implications and which is
right, although my inclination is per-package precisely because it avoids
the complexity of dealing with cycles. I asked Sam to send some details
about the cycles in k8s and docker repos.

- Sam had some concerns about depending so heavily on semver. He pointed
out that cockroachdb depends on pinned revs of x/text for collation tables
to ensure they don't change from release to release.

- Sam had some concerns about the scope of version exclusions: respecting
them only in the top level manifest is probably not good enough because it
means everyone using a package has to copy that package's exclusions to get
a good build. I agree that that's problematic and that manifest exclusions
in dependencies probably need to apply too (but still without a
satisfiability search).

- Sam reported some discussions with someone about Amazon's "version sets",
which are a bit like internal Google "components" and now Bazel WORKSPACE
files. It might be nice if that model can be expressed in manifests.

- Peter asked what about repos without semver tags, and my response was
that the behavior should degrade gracefully to today's, namely always use
the latest rev.

- Ed asked about branch designators, so that manifests can say "use this
branch." That's now on my list to investigate further.

- Sam asked about type leakage in general: if you have v1 and v2 of a
package in the same binary, how does the user understand what's going on
when the compiler refuses to pass a value of v1 type T to a function
expecting a v2 type T? I pointed out that the compiler already
automatically adds full import path context when the package names alone
are ambiguous, and we'd probably need to be careful to do the same here.

- Sam suggested dropping the section in the doc about the standard library,
to avoid speculative bikeshedding. (Even though my point in that paragraph
was to cut off just that, removing the section entirely might work better.)

- Ed pointed out that allowing multiple versions of a package is
problematic if you have a development model where you bump the major
version frequently: if there are v1 through v20 of your package, users
might not want all of them in their binaries. He also brought up a general
problem with people doing too much work at init time, and noted that having
many copies of a package exacerbates that.

- Peter said he wasn't convinced about needing multiple versions of a
package at all and wanted to push back on that idea. I raised the scenario
of having json v1 and introducing json v2, and suppose the official Amazon
APIs are using json v1 while the official Google Cloud APIs are using json
v2, and in both cases json doesn't show up in the API at all, it's just
purely an internal detail for preparing the wire format of the API
requests. The effect of disallowing multiple versions would be that a
single program can't use both the Amazon and Google Cloud APIs, because of
an internal implementation detail in those packages. The same thing comes
up for any similar utility library that might go through multiple major
versions. To me that's a show-stopper that must be avoided. I might have
read the room wrong, but to me people seemed generally receptive to this
example.

- Sam said that he was worried about "using this tool in anger" and that
the sketch assumes that people never stray from a "happy path", and if they
do then they will get stuck without any recourse. In contrast he seemed to
be saying that much of the complexity of dep and gps is meant to give
people more power in those situations, because "dependency management is
terrible and always will be."

- Ed raised a concern about per-package manifests being distributed
throughout a source tree and said he'd prefer having just one per project
(in the project root). I asked about lock files and whether those should be
per-package, since it seems that a lock file is fundamentally about
building a single target and if the lock file applies to all targets in a
project then it might not be well-specified. In general it seems to me that
we're boxed in a bit by the pre-existing package structure and may not be
able to impose repository-level ("project") constraints on existing usage.

- Peter raised a process concern, namely that there has already been an
extended documentation and feedback cycle and that he believes basically
everyone has been heard and that right now what we need instead is for
"people who know what they're talking about to lead". People seemed to
agree, and I do too. I clarified that I don't want to start a whole new
survey round but I do want to try to keep the community involved in seeing
the design of package management support in the go command take shape. I
don't expect to publish the public doc as a formal proposal, because it's
not. I might end up just posting something on my blog about "this is what
we're looking at" and just deal with feedback in whatever form it takes.

- Sam raised the general issue of the status of dep and in particular said
he had some misgivings about encouraging contributions to dep if it's not
going to last in its current form. I expanded a bit on what I wrote toward
the end of the doc, namely that dep is an important step in the process but
ultimately not the end result. It's critical for people to get experience
using (and to encourage them to use) semver tags and to help us learn what
works and doesn't for the eventual integration in the go command, but it's
equally important that integration is a full fitting into the entire go
command and not a new subcommand. I said that it's very important to keep
fixing bugs and issues where dep breaks, but it's probably not worthwhile
to design and add new dep features at this point. We know what dep is and
need to learn from it and use it as an interim tool. I also mentioned that
part of my plan for integration is to publish a go get-able copy of the go
command with the experimental integration for people to use with standard
toolchains (called 'xgo'). Go 1.9 will have the minimal latent support for
this in the compiler and linker, so that people can go get xgo and use it
for builds but know that they're still using a stable compiler, runtime,
and library set.

- Ed brought up the question of how Heroku and other systems that do deploy
as 'git push' should work. With vendoring, all the code can be in the git
repo, and also govendor's config at least has the flexibility of letting
Heroku store some otherwise-ignored config info in its config file, to
reduce the number of config files. That gets back a bit to per-project vs
per-package manifests. I had to run out at this point to go to another
meeting, but I want to follow up with Ed about this as plans firm up. It
would be nice to unify what Heroku needs and what App Engine needs,
including support within the go command. I had suggested that the 'build
plans' described in the doc might help but while App Engine has an explicit
deploy command, Heroku is only git push, so that makes things more
interesting. Definitely want to return to this when things are farther
along.

Russ