Ett 1788 solr fields by liseli · Pull Request #248 · hathitrust/babel

liseli · 2026-08-21T17:05:38Z

This PR is about the Jira ticket

The main goal was to split Solr 6/Solr 9 config files because Solr 9's schema removed the fields ctrlnum, fulgeographic, fullgenre, and hlb3, documentation here

The solution was to add ls/lib/Config/solr9/ as a copy of ls/lib/Config/ and remove the fields from there. The folder ls/lib/Config/ is untouched and is the default configuration to start Babel accessing to Solr 6.

A new ls_config_dir config key has been defined. By default, LSS_CONFIG_DIR=ls/lib/Config and lets ls/cgi/ls and LS::FacetConfig::get_rel_weights_from_yaml pick the right directory instead of hardcoding ls/lib/Config. To connect to Solr9, the Compose profile LSS_CONFIG_DIR was defined in apache.yml/perl.yml.

I've updated the README.md file with the commands to run against Solr 9, adding LSS_CONFIG_DIR.

I've tested these changes on both Solr 6 and Solr 9, and I get results. I'll listed the commands below, so you can reproduce them locally.

Testing Babel + Solr 6

docker compose --profile frontend build
docker compose --profile backend build

docker compose --profile frontend up
docker compose --profile backend up

Testing Babel + Solr 9

docker compose --profile frontend build
LSS_SOLR_HOST=solr-lss-dev-solr9 LSS_SOLR_CORE_B=core-x LSS_CONFIG_DIR=ls/lib/Config/solr9 docker compose --profile backend --profile solr9 build

docker compose --profile frontend up
LSS_SOLR_HOST=solr-lss-dev-solr9 LSS_SOLR_CORE_B=core-x LSS_CONFIG_DIR=ls/lib/Config/solr9 docker compose --profile backend --profile solr9 up

In Solr 9, you should index the sample data on the server. The process for indexing data is explained here. I've tested it, and it works, but you will need to clone the lss_solr_configs repository to your machine and follow the steps to run the script indexing_data.sh.

If in the service solr-lss-dev-solr9 (docker logs solr-lss-dev-solr9) you see the error Could not create a new core in /var/solr/data/core-x_shard1_replica_n1 as another core is already defined there. This happens because the persist volumes babel_solr1_data and babel_zoo1_data exist from a previous run, and the create_collection call collided with the leftover directory. There is a Jira ticket to fix this problem

Temporary solution: --> I'll create a task to fix this issue.

docker compose --profile solr9 rm -f solr-lss-dev-solr-solr9
docker volume rm babel_solr1_data
LSS_SOLR_HOST=solr-lss-dev-solr9 LSS_SOLR_CORE_B=core-x LSS_CONFIG_DIR=ls/lib/Config/solr9 docker compose --profile backend --profile solr9 up

Following up on this ticket, it's a good time to start thinking about testing. Let's consider how we can verify that the existing LS tests cover both Solr 6 and Solr 9, and whether we might need to add any extra unit tests. Of course, this will be part of a different task and epic. Please feel free to add any additional tickets that came up during your review of this PR and that you think we should work on.

I've created some tests with Claude support on ls/t/LS/Query/Facets.t

aelkiss

Queries are working locally for me with solr 9; I haven't tested locally with solr 6 yet.

I also still want to look some at the question of tests here.

I have some questions about some of the duplicated files, but the bigger question is when is the right time to try to disable interleave (A vs B) for solr 9 -- do we try to do that now, or do we wait until later? On the one hand, it may require fewer changes to keep it with interleave enabled than if we try to disable it. On the other, there are already some some workarounds here to make it work despite not having two different cores; this PR adds some additional workarounds, and I don't think we want to continue those workarounds indefinitely.

aelkiss · 2026-08-25T13:41:27Z

+
+side_by_side =0
+use_B_query= 0
+use_interleave=1


Does it make sense to try to turn off interleave entirely for solr 9, since we don't have a separate core-y set up there?

If use_interleave=1 works to disable interleave for solr9, that is something we should check

aelkiss · 2026-08-25T14:37:38Z

Do we need additional copies of the files under solr9 when they are the same as in the base Config directory (i.e. will they be found?). I think this is the case at least for langformat.yaml and bindings.pl.

If we do need them under solr9, does it make sense to make a symlink into the parent Config directory for them, rather than copying?

We might consider setting up a symlink; however, this method creates dependencies between Solr 6 and Solr 9 config files, so it’s important to keep track of these details in our documentation. Duplicating the files can also make the transition smoother when we’re ready to retire Solr 6, since it allows us to safely delete the folder when the time comes.

Note that Solr9/facetconfig.pl is still reading the original copy (edit: and Eugene has already pointed that out in a comment):

my $lang_format_file = $ENV{SDRROOT} . '/ls/lib/Config/langformat.yaml';

If we don't intend to use the Solr9 copy then that's argument for not having it. I would expect that data to be invariant no matter which Solr we are using.

As ls_config_dir is a MdpConfig key (ls/lib/Config/global.com), this variable is defined in the config object $C, and $C is not in scope inside facetconfig.pl because it only reaches variables from $ENV.

To keep things simple and aligned with my plan to remove all Solr 6 configurations once we're ready. I've removed the Solr 6 hardcoded path as follows.

Resolve langformat.yaml from this file's own directory (selected by the ls_config_dir config key) my $lang_format_file = dirname(__FILE__) . '/langformat.yaml';

Claude created a test to ensure that the identical file between Solr 6 and Solr 9 (AB_test_config, bindings.pl, langformat.yaml and jc_css_filelist.txt) do not change.

eumalin · 2026-08-25T15:45:55Z

Config is duplicated across two trees (ls/lib/Config and .../solr9) instead of one source with overrides. It may be causing a drift, such as sdrnum missing from the solr9 copy. It's worth thinking about consolidating them, it's not a priority/blocker, just a thought.

eumalin · 2026-08-25T15:36:26Z

+    - [author2, 50]
+    - [issn, 1]
+    - [isbn, 1]
+    - [oclc, 1]


Looks like [sdrnum, 1] is missing here, should it be added back?

Good catch!

eumalin · 2026-08-25T15:41:25Z

+#my $rel_weights_file_A = $ENV{SDRROOT} . '/ls/lib/Config/dismax_A.yaml';
+#my $rel_weights_file_B = $ENV{SDRROOT} . '/ls/lib/Config/dismax_B.yaml';
+
+my $lang_format_file = $ENV{SDRROOT} . '/ls/lib/Config/langformat.yaml';


Should this be using the new ls_config_dir variable?

@eumalin I answered this question in the comment above.

eumalin · 2026-08-25T15:42:12Z

 $C->set_object('MdpConfig', $config);

 # additional configuration for click_logging and A/B testing
 my $AB_test_config_filename = $ENV{SDRROOT} . '/ls/lib/Config/AB_test_config';


Should this be using the new ls_config_dir variable?

This is the A/B test file, which is schema-independent. For now, I'll keep it hardcoded. The logic is:

solr9/ holds every file (for the clean mv solr9/* up at cutover), but only the schema-bearing ones load from it: facetconfig.pl, dismax_A/B.yaml, plus langformat.yaml (via dirname(__FILE__), since it's facetconfig.pl's own data file). AB_test_config, bindings.pl, js_css_filelist.txt stay hardcoded to ls/lib/Config/ and the test enforces they match their solr9/ copies.

I'll add this information to the documentation

liseli · 2026-08-26T18:19:18Z

Queries are working locally for me with solr 9; I haven't tested locally with solr 6 yet.

I also still want to look some at the question of tests here.

I have some questions about some of the duplicated files, but the bigger question is when is the right time to try to disable interleave (A vs B) for solr 9 -- do we try to do that now, or do we wait until later? On the one hand, it may require fewer changes to keep it with interleave enabled than if we try to disable it. On the other, there are already some some workarounds here to make it work despite not having two different cores; this PR adds some additional workarounds, and I don't think we want to continue those workarounds indefinitely.

Let's try disabling the interleave test for Solr 9 later; it should be an optimization, probably in the user stories related to measuring relevance results. Right now, the focus is ensuring we can run queries on Solr 9 and Solr 6 in the production environment. I'll create a task to disable interleave. In this PR, I kept the interleave but pointed to only core-x for Solr 9, so we do not have to create core-y.

liseli · 2026-08-26T18:32:38Z

Config is duplicated across two trees (ls/lib/Config and .../solr9) instead of one source with overrides. It may be causing a drift, such as sdrnum missing from the solr9 copy. It's worth thinking about consolidating them, it's not a priority/blocker, just a thought.

I have duplicated the Config folders because some fields used in Solr 6 have been removed from Solr 9.

moseshll · 2026-08-27T19:35:19Z

  };
 };

+subtest 'solr9 facetconfig has incompatible fields removed' => sub {


Good to have these tests. They don't really belong here though. Really this is testing a component of FacetConfig.pm (which I will rant about a little bit later) so I would create a file ls/t/LS/FacetConfig.t and populate it with

use strict; use warnings; use Test::More; <this subtest> done_testing();

Then restore this file to its original state.

aelkiss · 2026-08-31T19:40:43Z

+    - [lccn, 1]
+    - [oclc, 1]
+    - [sdrnum, 1]
+    - [ht_id, 1]


I think this should be id, not ht_id. I got an error trying with beta-1 when doing an "all fields" advanced search complaining about ht_id -- I don't think it's there in solr 6 either, but I guess it doesn't give an error?

aelkiss

There were a few things where I wanted to make sure we had follow-up tasks noted; that's taken care of. This should be ready to go, although @liseli I would recommend squashing before merging.

… parameterization) parameterize LSS_SOLR core name for Solr 9 core-x-only routing ls: resolve uber.conf mbooks_solr_engines TODO (slip/index/sync-i depends on it) ls: remove dead Operation::Login entry from bindings.pl %g_operation_params document LSS_CONFIG_DIR for running against Solr 9 split Solr 6/Solr 9 facet config into ls/lib/Config/solr9/ and add ls_config_dir -replicationFactor is deprecated, so replace by --replicationFactor Temporarly solution to fix the error Solr 9 could not create a new core restore /solr context path in solr-lss-dev-solr9 healthcheck) Adding tests for solr 6 and solr 9 directory and Resolve langformat.yaml from this file's own directory Creating the file ls/t/LS/FacetConfig.t to handle test related to solr9 config files Replace ht_id by id in solr9 config directory

liseli requested review from aelkiss, eumalin and moseshll August 21, 2026 17:30

aelkiss reviewed Aug 25, 2026

View reviewed changes

eumalin reviewed Aug 25, 2026

View reviewed changes

aelkiss reviewed Aug 26, 2026

View reviewed changes

Comment thread common/docker/solr.yml

moseshll reviewed Aug 27, 2026

View reviewed changes

Comment thread ls/lib/LS/FacetConfig.pm

aelkiss reviewed Aug 31, 2026

View reviewed changes

aelkiss self-requested a review September 3, 2026 13:49

aelkiss approved these changes Sep 3, 2026

View reviewed changes

aelkiss mentioned this pull request Sep 3, 2026

ETT-1793 solr init script #250

Merged

liseli force-pushed the ETT-1788_SolrFields branch from 4b714d4 to b0e2d02 Compare September 3, 2026 19:08

liseli force-pushed the ETT-1788_SolrFields branch from b0e2d02 to 5745386 Compare September 3, 2026 19:33

liseli merged commit ff2106c into main Sep 3, 2026
3 checks passed

liseli deleted the ETT-1788_SolrFields branch September 3, 2026 20:43

Uh oh!

Conversation

liseli commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aelkiss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

moseshll Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eumalin commented Aug 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liseli Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

liseli commented Aug 26, 2026

Uh oh!

liseli commented Aug 26, 2026

Uh oh!

moseshll Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aelkiss left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liseli commented Aug 21, 2026 •

edited

Loading

moseshll Aug 27, 2026 •

edited

Loading

liseli Aug 26, 2026 •

edited

Loading

moseshll Aug 27, 2026 •

edited

Loading

aelkiss left a comment •

edited

Loading