Stabilize --extern flag without a path.

[ehuss]

This stabilizes the --extern flag without a path, implemented in #54116.

This flag is used to add a crate that may be found in the search path to the extern prelude. The intent of stabilizing this now is to change Cargo to emit this flag for proc_macro when building a proc-macro crate. This will allow the ability to elide extern crate proc_macro; for proc-macros, one of the few places where it is still necessary.

It is intended that Cargo may also use this flag for other cases in the future as part of the std-aware work. There will likely be some kind of syntax where users may declare dependencies on other crates (such as alloc), and Cargo will use this flag so that they may be used like any other crate. At this time there are no short-term plans to use it for anything other than proc-macro.

This will not help for non-proc-macro crates that use proc_macro, which I believe is not too common?

An alternate approach for proc-macro is to use the meta crate, but from my inquiries there doesn't appear to be anyone interested in pushing that forward. The meta crate also doesn't help with things like alloc or test.

cc #57288

[Centril]

Is this the only test?

[ehuss]

extern-flag-fun is not actually testing this feature. It is a 5-year old test from when --extern was first added. The removed --extern hello line was originally there to verify that it would error due to the missing path. When pathless --extern was added, the error changed to "missing -Z unstable-options". With this PR, the hello line no longer fails (and thus breaks the test), due to -L flags injected by the makefile. I figured it no longer fit the spirit of the test.

I have added some tests.

[Centril]

Do we have tests ensuring that sysroot crates, e.g. private rustc details emit an error when used on a stable compiler? (Also when an explicit path is used to them, not just a search.)

[petrochenkov]

What happens if both --extern CRATENAME=PATH and --extern CRATENAME are specified for the same CRATENAME? It would be nice to document that.
AFAIR, multiple --extern CRATENAME=PATHs with the same CRATENAME are possible if they lead to different kinds of libraries, like rlib and dylib.

[ehuss]

There are about 30 tests doing various things with pathless --extern (search for // compile-flags:.*--extern).

I have added some more specific tests:

• run-make-fulldeps/extern-flag-pathless: Shows what happens when you mix pathless with path --extern flags. Also demonstrates the preference of rlib over dylib.
• ui-fulldeps/pathless-extern-unstable.rs: Shows --extern rustc is not allowed.
• ui/pathless-extern-ok.rs: Basic test for a sysroot crate.

I don't have a good intuition of how sophisticated rustc tests should be. They are also a bit disorganized, so I didn't know where to stick them, or which style would be preferred. I think the dylib tests should be safe from a cross platform standpoint (I believe they only run on Linux?).

I updated the documentation with some more detail. I never really know how detailed rustc docs should be. I intentionally left the prefer-dynamic algorithm vague. It is described somewhat in src/librustc_metadata/dependency_format.rs. I figure since it is complex, and possibly subject to change, it may not be worthwhile. I can add more or remove some if desired.

[withoutboats]

We discussed this in the lang team meeting. In general, we are comfortable with stabilizing this feature. We would like some more commentary from other people with more knowledge of build systems, the sysroot, etc (lang team members with more relevant experience here weren't available today), mainly about what possible hazards there could be with this feature.

In terms of the larger goal this is trying to solve (eliminating the remaining use cases for extern crate), we had some other thoughts. This is really something that has been dropped on the floor, and no one seems available to shepherd it. It seems good to make proc_macro available in crates with proc-macro marked in lib, and this plus the associated change to cargo seems like a great solution to that problem. In the long term, we would like to reorganize the proc_macro APIs under meta and deprecate proc_macro, but - again - no one is shepherding that (as you commented).

However, it's not clear if this will be a step to solving the no_std and test problems, which have some important differences (most importantly, currently there's no clear way the need for these crates is marked in the toml). We think a full RFC process would be necessary to figure out the interface to access these crates without using extern crate.

[ehuss]

We think a full RFC process would be necessary to figure out the interface to access these crates without using extern crate.

Absolutely. That is the intent of rust-lang/wg-cargo-std-aware#5 to produce an RFC with the necessary Cargo.toml syntax to make these dependencies explicit, and is the next major milestone I plan to tackle.

[Centril]

Is this achieved by whitelist?

In a sense, yes. The crate must be marked with #![stable], otherwise the -Zforce-unstable-if-unmarked makes it unstable.

Ah... so just for my education, what happens if -Z force-unstable-if-unmarked isn't there? (And it cannot be there on stable when you are not using cargo since it's a -Z flag?)

It will successfully link whatever library is in the given path. That library will be added to the extern prelude with the identifier "alloc".

Isn't that a problem then if you are not using cargo since it would allow you to use unstable stuff on stable? I'm rather surprised that -Z force-unstable-if-unmarked isn't the default and that you need a flag to opt-out for sysroot crates.

Sorry, I'm not understanding the question.

On stable rustc, you cannot access any crates that are marked "unstable". Either from --extern foo or --extern foo=path/to/sysroot/libfoo.rlib or extern crate foo; or -L path/to/unstable/crates.

On nightly, you can include the attribute #![feature(rustc_private)] to override this check.

All of the sysroot crates are built with the -Zforce-unstable-if-unmarked flag.

The opt-in nature is just saying #![stable] crates are OK to access, even with the -Zforce-unstable-if-unmarked flag (that is the "if unmarked" part). I believe the only crates marked stable are core, std, alloc, and proc_macro.

In theory, anyone can compile any crate using unstable features using nightly, and then load those crates from stable. In practice, you can't (due to rustc version checking) or clearly something you shouldn't do (like set RUSTC_BOOTSTRAP env var).

Can you say more how you think that can be circumvented?

All of the sysroot crates are built with the -Zforce-unstable-if-unmarked flag.

Oh! -- I missed this part. I thought you meant that when the sysroot crate is used, unless there is a -Zforce..., then it won't be gated... but I see now that this is about when the crate is built, not used. Now it clicks. Maybe this conversation should be distilled into the rustc guide?

Done: rust-lang/rustc-dev-guide#462

Absolutely. That is the intent of rust-lang/wg-cargo-std-aware#5 to produce an RFC with the necessary Cargo.toml syntax to make these dependencies explicit, and is the next major milestone I plan to tackle.

As a note, one reason I think we have not been enthusiastic on this is for the primary use case - core, alloc - there's a lot of good reason to deprecate these crates and move to a system that works more like feature flags on a single crate (so we can add impls without orphan issues, for example). However, this is a big and complicated refactor of the standard library and no one is driving it.

@rfcbot fcp merge

Dear compiler team,

I propose that we merge this PR. It adds a --extern foo flag which adds a crate foo from the syspath (in contrast to --extern foo=path which adds the crate foo from a specific path). The lang team has signed off on the general concept.

Team member @nikomatsakis has proposed to merge this. The next step is review by the rest of the tagged team members:

• @Zoxc
• @cramertj
• @eddyb
• @estebank
• @michaelwoerister
• @nagisa
• @nikomatsakis
• @oli-obk
• @petrochenkov
• @pnkfelix
• @varkor

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

I just realized that this does not support crate renaming (the equivalent of extern crate test as std_test). Some options:

1. Delay this stabilization until it is supported.
2. Add support later through some extension of the existing CLI syntax.
3. Don't support renaming.

It looks like it would be relatively easy to add. I'm curious what anyone else thinks. I'd also like to hear if anyone would have any ideas on the cli syntax.

I think I'd like to lean towards 1 or 2, as I think it would be an odd wart not to support it. And if others agree, maybe waiting would be prudent?

2 is ok, IMO.

The semantics are clear

--extern foo=PATH // search by exact path `PATH` and add to extern prelude as `foo`
--extern foo???bar // search by name `bar` in the search directories and add to extern prelude as `foo`
--extern foo // sugar for `--extern foo???foo`

we only need to review the syntactic space so that --extern foo???bar is never ambiguous with --extern foo=PATH and perhaps with --extern:modifier foo[=PATH|???bar].

Do we want to guarantee that? Does any tools rely on that? (The "rlib ->rmeta -> dylib" order for metadata.)

Hypothetically, what if we want to change it to "rmeta -> rlib -> dylib"?
rmeta is smaller, easier to obtain, perhaps faster to read, etc - same reasons why rlib is preferred to dylib, basically. Perhaps we can exploit that somehow.
Then maybe it better stay unspecified in that case?

Sounds good, I have removed it. I am not aware of any tools relying on that behavior, and I think it would be a little strange to do so.

Ping from triage:
This PR seems to be waiting on review @frewsxcv @petrochenkov
Cc: @ehuss
Thanks

I'm not familiar with any of this code; can someone else review it?

Ping @Zoxc @michaelwoerister, and @nagisa.

r? @eddyb or maybe @petrochenkov

Please fix the id duplicate then it can get r+ again.

@bors: r-

@bors r=eddyb

Strange that github/bors did not have a merge conflict, since I got one locally.

📌 Commit ee459c6 has been approved by eddyb

⌛ Testing commit ee459c6 with merge d257440...

☀️ Test successful - checks-azure
Approved by: eddyb
Pushing d257440 to master...

📣 Toolstate changed by #64882!

Tested on commit d257440.
Direct link to PR: #64882

💔 rls on linux: test-pass → test-fail (cc @Xanewok, @rust-lang/infra).