Private repositories and proxies: URL credentials, proxy auth, mirror-aware lookups, docs

README.md

@@ -186,10 +186,8 @@ cellar get-external org.typelevel:cats-core_3:latest cats.Monad
 | `-v`, `--verbose` | all except `telemetry` | Log progress and warnings to stderr |
 | `--debug` | all except `telemetry` | Log detailed diagnostics and stack traces to stderr |
 
-Private repositories: credentials from `~/.config/coursier/credentials.properties` (or
-`COURSIER_CONFIG_DIR`) and the `COURSIER_CREDENTIALS` environment variable are read the same way
-the coursier CLI, sbt and Mill read them, and applied to every repository whose host matches.
-See the [coursier credentials docs](https://get-coursier.io/docs/other-credentials).
+Private repositories need credentials in coursier's format — see
+[Private repositories](#private-repositories).
 
 Diagnostics are written to stderr, never stdout, so `--verbose` and `--debug` are safe to use
 when piping output into a prompt. Set `CELLAR_LOG=verbose` or `CELLAR_LOG=debug` to get the same
@@ -319,23 +317,124 @@ maven {
 
 Maven Central and Ivy local stay available, and CLI `-r` values are appended to the configured list.
 Lists replace rather than merge, so a project-level `maven.repositories` overrides the user-level
-list and `repositories = []` clears it for that project. Configuration holds URLs only —
-credentials come from coursier, as described above.
+list and `repositories = []` clears it for that project. For credentials, see
+[Private repositories](#private-repositories).
 
 Or via environment: `CELLAR_SBT_BINARY=sbtn cellar get --module core cats.Monad`
 
-Trust a TLS-intercepting proxy's CA. The native binary ignores `JAVA_TOOL_OPTIONS`, so set the
-trust store in `~/.cellar/cellar.conf` (same format as scala-cli's `java.properties`):
+### Private repositories
+
+Project-aware commands get the classpath from your build tool, so they use whatever credentials
+the build already has.
+
+External commands resolve with coursier and use exactly the credentials coursier reads — the same
+ones the coursier CLI and scala-cli use. cellar adds no sources of its own and never reads
+`~/.sbt/.credentials` or Gradle properties.
+
+#### Where coursier finds credentials
+
+1. `COURSIER_CREDENTIALS` (or `-Dcoursier.credentials`): inline entries, or an absolute path to a
+   properties file. When set, the files in 2. are ignored.
+2. `credentials.properties` and `credentials/*.properties` in the coursier config directory:
+   `~/.config/coursier` on Linux, `~/Library/Application Support/Coursier` on macOS, or
+   `COURSIER_CONFIG_DIR`.
+3. `repositories.credentials` in the scala-cli config (`scala-cli config repositories.credentials …`).
+4. Mirrors in Maven's `~/.m2/settings.xml`, with the credentials of the `<server>` sharing the
+   mirror's `id`. `COURSIER_MAVEN_SETTINGS=false`
+   turns this off; `COURSIER_MAVEN_SETTINGS=/path/to/settings.xml` picks another file.
+5. Credentials in a repository URL, see [below](#credentials-in-the-repository-url).
+
+`credentials.properties`:
+
+```properties
+company.host=artifactory.company.com
+company.username=alice
+company.password=s3cret
+# Optional:
+company.realm=Artifactory Realm
+company.https-only=true
+```
+
+Inline, one entry per line: `artifactory.company.com alice:s3cret`, or
+`artifactory.company.com(Artifactory Realm) alice:s3cret` with a realm.
+
+Credentials from 1.–3. apply to every repository on a matching host, but are held back until the
+server rejects an anonymous request with a `4xx` (`401`, `403` or `404`). Most servers do; Azure
+Artifacts is the known exception. Credentials from 4. and 5. are sent with the first request.
+
+Coming from sbt, an `~/.sbt/.credentials` file maps one-to-one:
+
+| sbt `.credentials` | coursier `credentials.properties` |
+|--------------------|-----------------------------------|
+| `realm=…`          | `<name>.realm=…`                  |
+| `host=…`           | `<name>.host=…`                   |
+| `user=…`           | `<name>.username=…`               |
+| `password=…`       | `<name>.password=…`               |
+
+#### Credentials in the repository URL
+
+Azure Artifacts answers anonymous requests with `203` and a sign-in page rather than a `4xx`, so
+held-back credentials never reach it. Put them in the repository URL instead:
+
+```hocon
+maven {
+  repositories = ["https://user:"${?AZURE_ARTIFACTS_TOKEN}"@pkgs.dev.azure.com/org/_packaging/feed/maven/v1"]
+}
+```
+
+`${?…}` reads the environment, so the token stays out of the file and commands keep working where
+it is unset. The same URL works with `-r`. URL credentials are sent over HTTPS only, unless the
+URL itself is `http://`, and cellar never prints them. They cannot contain `/`, `?`, `#` or `@`
+(coursier takes them verbatim, so percent-encoding does not help); use a credentials file then.
+
+#### Per-project credentials
+
+Coursier never looks for credentials in the project directory. Either commit the URL form above
+to `.cellar/cellar.conf`, with the token taken from the environment, or point
+`COURSIER_CREDENTIALS` at a file outside version control — an absolute path, as anything else is
+read as inline entries:
+
+```sh
+export COURSIER_CREDENTIALS="$PWD/.coursier-credentials.properties"   # e.g. from .envrc; gitignore it
+```
+
+#### Proxies
+
+Proxy settings are JVM system properties. The native binary ignores `JAVA_TOOL_OPTIONS`, so set
+them in `~/.cellar/cellar.conf` (same format as scala-cli's `java.properties`):
 
 ```hocon
 java {
   properties = [
-    "javax.net.ssl.trustStore=/path/to/cacerts",
-    "javax.net.ssl.trustStorePassword=changeit"
+    "https.proxyHost=proxy.company.com", "https.proxyPort=3128",
+    "http.proxyHost=proxy.company.com", "http.proxyPort=3128",
+    "http.nonProxyHosts=localhost|*.company.com",
+    # Only for a proxy that asks for credentials:
+    "https.proxyUser=alice", "https.proxyPassword="${?PROXY_PASSWORD},
+    # Only for a TLS-intercepting proxy:
+    "javax.net.ssl.trustStore=/path/to/cacerts", "javax.net.ssl.trustStorePassword=changeit"
   ]
 }
 ```
 
+To go through a company mirror instead of Maven Central, configure the mirror in coursier
+(`mirror.properties`, see the [coursier docs](https://get-coursier.io/docs/other-proxy)) or in
+Maven's `settings.xml`. cellar does not read proxy settings from `settings.xml`.
+
+#### Troubleshooting
+
+A failed resolution lists every location coursier tried:
+
+| Shown under `Tried:`                        | Meaning                                                                   |
+|---------------------------------------------|---------------------------------------------------------------------------|
+| `unauthorized: <url> (<realm>)`             | No credentials for that host, or the server rejected them                 |
+| `forbidden: <url>`                          | No credentials for that host, or they lack access                         |
+| `checksum format error: …`                  | The server sent a page instead of the artifact (e.g. Azure Artifacts' sign-in page): [put the credentials in the URL](#credentials-in-the-repository-url) |
+| `… 407 …` / `Unable to tunnel through proxy` | The proxy wants credentials: set `http(s).proxyUser` and `http(s).proxyPassword` |
+
+To see which credentials coursier sends to each URL, set `COURSIER_HTTP_DEBUG=true`. It names the
+user, never the password.
+
 ## Telemetry
 
 Cellar collects **anonymous usage telemetry** to help improve the tool. It is opt-in. On first use cellar asks you to choose before running:

build.mill

@@ -205,14 +205,18 @@ object cli extends CellarPublishModule with NativeImageModule {
   //
   // GC-bound builds on 7 GB macOS runners outlast the default 10-minute deadlock watchdog.
   // Unsafe access is fixed at build time; otherwise every run warns about Scala's `LazyVals`.
+  // So is the JDK's list of auth schemes refused for HTTPS proxy tunnels, which makes the runtime
+  // opt-in in `JvmProperties` a no-op here; Basic must be allowed at build time for a proxy asking
+  // for credentials. Without a proxy user nothing answers the challenge either way.
   def nativeImageOptions = Seq(
     "--no-fallback",
     "--enable-url-protocols=http,https",
     "-H:+ReportExceptionStackTraces",
     "-H:+UnlockExperimentalVMOptions",
     "-H:DeadlockWatchdogInterval=30",
     "-H:-UnlockExperimentalVMOptions",
-    "-J--sun-misc-unsafe-memory-access=allow"
+    "-J--sun-misc-unsafe-memory-access=allow",
+    "-Djdk.http.auth.tunneling.disabledSchemes="
   )
 
   def mvnDeps = Seq(

cli/src/cellar/cli/CellarApp.scala

@@ -129,7 +129,7 @@ object CellarApp extends ProfilingIOApp:
       case Left(error) =>
         IO.blocking(System.err.println(error)).as(ExitCode.Error)
       case Right(properties) =>
-        IO(properties.foreach(sys.props.getOrElseUpdate(_, _))) *> runCommand(args)
+        JvmProperties.install(properties) *> runCommand(args)
 
   private def runCommand(args: List[String]): IO[ExitCode] =
     val versionOpt = Opts

cli/src/cellar/cli/JvmProperties.scala

@@ -0,0 +1,23 @@
+package cellar.cli
+
+import cats.effect.IO
+import coursier.proxy.SetupProxy
+
+object JvmProperties:
+
+  /** Sets each `java.properties` entry not already set (e.g. by `-D`) and enables proxy
+    * authentication from `http(s).proxyUser` / `http(s).proxyPassword`.
+    *
+    * Must run before the first HTTP connection: the JDK reads its tunneling settings once.
+    */
+  def install(properties: List[(String, String)]): IO[Unit] =
+    IO {
+      properties.foreach(sys.props.getOrElseUpdate(_, _))
+      // coursier answers a proxy's challenge only when the protocol the JDK reports matches these,
+      // and it defaults the https one to "https". The JDK only ever speaks plain HTTP to a proxy,
+      // so a setup with just `https.proxy*` would never be answered.
+      List("http.proxyProtocol", "https.proxyProtocol").foreach(sys.props.getOrElseUpdate(_, "http"))
+      // The JDK ignores the proxy user and password on its own and refuses Basic auth for HTTPS
+      // tunnels; this installs what the coursier launcher does, and only when a proxy user is set.
+      SetupProxy.setupAuthenticator(): Unit
+    }

cli/src/cellar/cli/RepositoryUrl.scala

@@ -1,13 +1,17 @@
 package cellar.cli
 
 import coursier.MavenRepository
+import coursier.parse.RepositoryParser
 
 import java.net.URI
 import scala.util.Try
 
 /** `MavenRepository(...)` accepts any string verbatim, so a typo such as a missing scheme survives
   * until resolution fails — and it then fails as a coordinate error, pointing at the coordinate
   * rather than at the repository. Rejecting the URL up front keeps the blame where it belongs.
+  *
+  * Credentials in the URL (`https://user:token@host/...`) become the repository's own and are sent
+  * with the first request; error messages mask them.
   */
 object RepositoryUrl:
 
@@ -18,18 +22,35 @@ object RepositoryUrl:
   /** Left carries the reason only; the caller prefixes it with where the URL came from. */
   def parse(raw: String): Either[String, MavenRepository] =
     val trimmed = raw.trim
+    val shown   = masked(trimmed)
+    // coursier does not URL-decode credentials, so these characters cannot be written in any form.
+    val credentialsHint =
+      if shown == trimmed then ""
+      else "; credentials in a URL cannot contain '/', '?', '#' or '@', use COURSIER_CREDENTIALS instead"
     if trimmed.isEmpty then Left("the URL is empty")
     else
       Try(URI(trimmed)).toEither.left
-        .map(_ => s"'$raw' is not a valid URL")
+        .map(_ => s"'$shown' is not a valid URL$credentialsHint")
         .flatMap { uri =>
           Option(uri.getScheme).map(_.toLowerCase) match
             case None =>
-              Left(s"'$raw' has no scheme (expected one of $expected)")
+              Left(s"'$shown' has no scheme (expected one of $expected)")
             case Some(scheme) if !supportedSchemes.contains(scheme) =>
-              Left(s"'$raw' has unsupported scheme '$scheme' (expected one of $expected)")
+              Left(s"'$shown' has unsupported scheme '$scheme' (expected one of $expected)")
             case Some(scheme) if scheme != "file" && Option(uri.getHost).forall(_.isEmpty) =>
-              Left(s"'$raw' has no host")
+              Left(s"'$shown' has no host$credentialsHint")
             case Some(_) =>
-              Right(MavenRepository(trimmed))
+              // coursier's own message would echo the password; a scheme checked above always
+              // parses to a Maven repository.
+              RepositoryParser.repository(trimmed).left
+                .map(_ => s"'$shown' is not a valid URL$credentialsHint")
+                .map { case maven: MavenRepository => maven }
         }
+
+  /** Hides everything between the scheme and the last `@`. A token may itself contain `/`, `@` or
+    * `#`, so the usual delimiters cannot tell where the credentials end.
+    */
+  private def masked(url: String): String =
+    val schemeEnd = url.indexOf("://")
+    val at        = url.lastIndexOf('@')
+    if schemeEnd >= 0 && at > schemeEnd then url.take(schemeEnd + 3) + "****" + url.drop(at) else url

cli/test/src/cellar/cli/PrivateRepositoryTest.scala

@@ -0,0 +1,110 @@
+package cellar.cli
+
+import cats.effect.{IO, Resource}
+import cellar.{CoursierFetchClient, MavenCoordinate}
+import com.sun.net.httpserver.{HttpExchange, HttpServer}
+import weaver.SimpleIOSuite
+
+import java.net.{Authenticator, InetAddress, InetSocketAddress}
+import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Base64
+import java.util.concurrent.ConcurrentLinkedQueue
+import scala.jdk.CollectionConverters.*
+
+/** Fetches through real HTTP servers standing in for the private-repository setups the README documents. */
+object PrivateRepositoryTest extends SimpleIOSuite:
+
+  // The proxy tests set JVM-wide system properties and the default Authenticator.
+  override def maxParallelism = 1
+
+  private val coord   = MavenCoordinate("org.example.private", "lib", "1.0.0")
+  private val dir     = "/org/example/private/lib/1.0.0/"
+  private val content = Map(
+    s"${dir}lib-1.0.0.pom" ->
+      "<project><modelVersion>4.0.0</modelVersion><groupId>org.example.private</groupId><artifactId>lib</artifactId><version>1.0.0</version></project>",
+    s"${dir}lib-1.0.0.jar" -> "not inspected"
+  )
+
+  // coursier caches by URL, credentials' user included; a run reusing an earlier URL would be
+  // answered from the cache without reaching the servers under test.
+  private val run = java.util.UUID.randomUUID().toString.take(8)
+
+  private def basic(user: String, password: String) =
+    "Basic " + Base64.getEncoder.encodeToString(s"$user:$password".getBytes(UTF_8))
+
+  private def respond(exchange: HttpExchange, code: Int, body: String = ""): Unit =
+    val bytes = body.getBytes(UTF_8)
+    exchange.sendResponseHeaders(code, if bytes.isEmpty then -1 else bytes.length.toLong)
+    if bytes.nonEmpty then exchange.getResponseBody.write(bytes)
+    exchange.close()
+
+  private def serve(exchange: HttpExchange): Unit =
+    content.get(exchange.getRequestURI.getPath).fold(respond(exchange, 404))(respond(exchange, 200, _))
+
+  /** Starts `handle` on a free port; yields the port. */
+  private def server(handle: HttpExchange => Unit): Resource[IO, Int] =
+    Resource
+      .make(IO.blocking {
+        val server = HttpServer.create(InetSocketAddress("127.0.0.1", 0), 0)
+        server.createContext("/", exchange => handle(exchange))
+        server.start()
+        server
+      })(server => IO.blocking(server.stop(0)))
+      .map(_.getAddress.getPort)
+
+  test("URL credentials reach a feed that answers anonymous requests with a 203 sign-in page"):
+    val anonymous = ConcurrentLinkedQueue[String]()
+    val feed = server { exchange =>
+      if exchange.getRequestHeaders.getFirst("Authorization") == basic(s"alice-$run", "s3cret") then serve(exchange)
+      else
+        anonymous.add(exchange.getRequestURI.getPath): Unit
+        respond(exchange, 203, "<html>Sign in</html>")
+    }
+    feed.use { port =>
+      RepositoryUrl.parse(s"http://alice-$run:s3cret@127.0.0.1:$port/").fold(
+        err => IO.pure(failure(err)),
+        repo =>
+          CoursierFetchClient.fetchPom(coord, Seq(repo)).map { pom =>
+            expect(pom.isDefined) and expect.eql(Nil, anonymous.asScala.toList)
+          }
+      )
+    }
+
+  test("an authenticating proxy is passed the proxy user and password"):
+    val proxied = ConcurrentLinkedQueue[String]()
+    // A forward proxy is sent the full target URL; this one serves the repository itself.
+    val proxy = server { exchange =>
+      if exchange.getRequestHeaders.getFirst("Proxy-Authorization") == basic("proxyuser", "proxypw") then
+        proxied.add(exchange.getRequestURI.toString): Unit
+        serve(exchange)
+      else
+        exchange.getResponseHeaders.add("Proxy-Authenticate", """Basic realm="corp-proxy"""")
+        respond(exchange, 407)
+    }
+    val properties = List("http.proxyHost", "http.proxyPort", "http.proxyUser", "http.proxyPassword")
+    proxy.use { port =>
+      val install = JvmProperties.install(
+        properties.zip(List("127.0.0.1", port.toString, "proxyuser", "proxypw"))
+      )
+      val repo = RepositoryUrl.parse(s"http://repo-$run.example.invalid/").fold(err => throw AssertionError(err), identity)
+      (install *> CoursierFetchClient.fetchPom(coord, Seq(repo)))
+        .guarantee(IO(properties.foreach(sys.props.remove)))
+        .map { pom =>
+          expect(pom.isDefined) and
+            expect(proxied.asScala.exists(_.startsWith(s"http://repo-$run.example.invalid/org/example/private/lib/")))
+        }
+    }
+
+  test("a proxy configured only through https.proxy* properties is answered too"):
+    val properties = List("https.proxyHost", "https.proxyPort", "https.proxyUser", "https.proxyPassword")
+    val install    = JvmProperties.install(properties.zip(List("127.0.0.1", "3128", "proxyuser", "proxypw")))
+    // The JDK reports "http" for any proxy challenge, an HTTPS tunnel's included.
+    val challenge = IO(
+      Authenticator.requestPasswordAuthentication(
+        "127.0.0.1", InetAddress.getLoopbackAddress, 3128, "http", "corp-proxy", "basic", null,
+        Authenticator.RequestorType.PROXY
+      )
+    )
+    (install *> challenge)
+      .guarantee(IO((properties ++ List("http.proxyProtocol", "https.proxyProtocol")).foreach(sys.props.remove)))
+      .map(auth => expect(auth != null && auth.getUserName == "proxyuser", String.valueOf(auth)))

cli/test/src/cellar/cli/RepositoryUrlTest.scala

@@ -38,3 +38,25 @@ object RepositoryUrlTest extends FunSuite:
 
   test("rejects a string that is not a URL at all"):
     reason("not a url")(message => expect(message.contains("not a valid URL")))
+
+  test("moves URL credentials onto the repository, sent up front and over https only"):
+    RepositoryUrl.parse("https://alice:s3cret@pkgs.example.com/maven").fold(
+      err => failure(s"expected the URL to be accepted, got: $err"),
+      repo =>
+        repo.authentication.fold(failure("expected credentials")) { auth =>
+          expect.eql("https://pkgs.example.com/maven", repo.root) and
+            expect.eql((Some("alice"), Some("s3cret")), (auth.userOpt, auth.passwordOpt)) and
+            expect(!auth.optional && auth.httpsOnly)
+        }
+    )
+
+  test("never echoes URL credentials in an error, whatever characters they contain"):
+    def masked(raw: String, secret: String) =
+      reason(raw)(message => expect(message.contains("****@") && !message.contains(secret), message))
+    masked("ftp://alice:s3cret@pkgs.example.com/maven", "s3cret") and
+      masked("https://alice:ab/cd@pkgs.example.com/maven", "ab/cd") and
+      masked("https://alice:p@ss@pkgs.example.com/maven", "p@ss") and
+      masked("ftp://ghp_token@pkgs.example.com/maven", "ghp_token")
+
+  test("explains that credentials with URL delimiters cannot go in a URL"):
+    reason("https://alice:ab/cd@pkgs.example.com/maven")(message => expect(message.contains("COURSIER_CREDENTIALS"), message))