Skip to content

Adding solr cloud service to babel #239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
liseli merged 9 commits into from
Aug 13, 2026
Merged

Adding solr cloud service to babel #239

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8d254d0
Adding solr cloud service to babel
liseli Jul 9, 2026
3e4fdbd
Declare Zookeeper dependency when Solr is extended
liseli Jul 10, 2026
868ba27
Update solr.yml to start the Solr server using the image with Solr 9
liseli Jul 17, 2026
2820bcf
Removed uncommented lines on docker-compose
liseli Jul 21, 2026
d293c9c
Set up Solr 6 and Solr 9 in docker environment
liseli Aug 11, 2026
6d31165
add add_basic_auth to send Authorization header to ls has access to S…
liseli Aug 12, 2026
77eba67
Pass `Context` object `$C` to `__get_request_object` so that mdp-conf…
moseshll Aug 13, 2026
2699973
adding the context to __get_request_object and LSS_SOLR_BASIC_AUTH to…
liseli Aug 13, 2026
014ebfc
Document Solr credentials in the config files
liseli Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@ In your workdir:
docker compose --profile frontend build
docker compose --profile backend build
```
If you want to build the application with Solr 9, you can use the `solr9` and `backend` profiles:

```
LSS_SOLR_HOST=solr-lss-dev-solr9 docker compose --profile backend --profile solr9 build
```

## Step 4: Build assets

Build the CSS and JavaScript for `firebird-common` and `pt`:
Expand Down Expand Up @@ -57,6 +63,12 @@ docker compose --profile frontend up
docker compose --profile backend up
```

If you want to build the application with Solr 9, you can use the `solr9` and `backend` profiles:

```
LSS_SOLR_HOST=solr-lss-dev-solr9 docker compose --profile backend --profile solr9 up
```

In your browser:

* catalog: `http://localhost:8080/Search/Home`
Expand Down
7 changes: 7 additions & 0 deletions apache/000-default.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,13 @@ Listen 8080

SetEnv PTSEARCH_SOLR http://solr-ptsearch:8983/solr/ptsearch
SetEnv PTSEARCH_SOLR_BASIC_AUTH ""

# docker-compose.yml/apache.yml set the variables LSS_SOLR_ENGINE_URL_A/LSS_SOLR_ENGINE_URL_B with
# solr URL. We should add these variables here so the CGI process can access them.
# Use PassEnv to pass the environment variables to the CGI process so we can manage both Solr 6 and Solr 9 URLs
PassEnv LSS_SOLR_ENGINE_URL_A
PassEnv LSS_SOLR_ENGINE_URL_B
PassEnv LSS_SOLR_BASIC_AUTH

<Directory "/htapps/babel">
Options Indexes FollowSymLinks
Expand Down
4 changes: 4 additions & 0 deletions common/docker/apache.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ services:
environment:
- SDRROOT=/htapps/babel
- SDRDATAROOT=/sdr1
- LSS_SOLR_ENGINE_URL_A=http://${LSS_SOLR_HOST:-solr-lss-dev}:8983/solr/core-x
- LSS_SOLR_ENGINE_URL_B=http://${LSS_SOLR_HOST:-solr-lss-dev}:8983/solr/core-y
# This is the basic auth for Solr. The default 'admin:solrRocks' credential
- LSS_SOLR_BASIC_AUTH=YWRtaW46c29sclJvY2tz

@aelkiss aelkiss Aug 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be worth a comment that this is the default 'admin:solrRocks' credential

healthcheck:
interval: 5s
timeout: 10s
Expand Down
4 changes: 4 additions & 0 deletions common/docker/perl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ services:
- MARIADB_USER=ht_web
- HTDEV=test
- PERL5LIB=/htapps/babel/mdp-lib
- LSS_SOLR_ENGINE_URL_A=http://${LSS_SOLR_HOST:-solr-lss-dev}:8983/solr/core-x
- LSS_SOLR_ENGINE_URL_B=http://${LSS_SOLR_HOST:-solr-lss-dev}:8983/solr/core-y
# This is the basic auth for Solr. The default 'admin:solrRocks' credential
- LSS_SOLR_BASIC_AUTH=YWRtaW46c29sclJvY2tz
depends_on:
mysql-sdr:
condition: service_healthy
Expand Down
55 changes: 48 additions & 7 deletions common/docker/solr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,56 @@ services:
# This is another core service
image: solr:6.6.6-alpine
ports:
- "8983:8983"
- "8983:8983"
user: ${CURRENT_USER}
volumes:
- ${BABEL_HOME}/lss_solr_configs/solr6_standalone/lss-dev/core-x:/opt/solr/server/solr/core-x
- ${BABEL_HOME}/lss_solr_configs/solr6_standalone/lss-dev/core-y:/opt/solr/server/solr/core-y
- ${BABEL_HOME}/lss_solr_configs/solr6_standalone:/opt/lss_solr_configs
- ${BABEL_HOME}/lss_solr_configs/solr6_standalone/lib:/opt/solr/server/solr/lib
- ${BABEL_HOME}/logs/solr:/opt/solr/server/logs
- ${BABEL_HOME}/lss_solr_configs/solr6_standalone/lss-dev/core-x:/opt/solr/server/solr/core-x
- ${BABEL_HOME}/lss_solr_configs/solr6_standalone/lss-dev/core-y:/opt/solr/server/solr/core-y
- ${BABEL_HOME}/lss_solr_configs/solr6_standalone:/opt/lss_solr_configs
- ${BABEL_HOME}/lss_solr_configs/solr6_standalone/lib:/opt/solr/server/solr/lib
- ${BABEL_HOME}/logs/solr:/opt/solr/server/logs
healthcheck:
<<: *healthcheck-defaults
test: "ping -c 1 solr-lss-dev:8983"

solr-lss-dev-solr-solr9:
image: ghcr.io/hathitrust/full-text-search-cloud:solr9-shards-docker
container_name: solr-lss-dev-solr9
ports:
- "8981:8983"
environment:
- ZK_HOST=zoo1:2181
- SOLR_OPTS=-XX:-UseLargePages
- SOLR_MODULES=analysis-extras
volumes:
- solr1_data:/var/solr
command: [ "sh", "-c", "solr-foreground & sleep 150 && export SOLR_AUTHENTICATION_OPTS=-Dbasicauth=admin:solrRocks && solr create_collection -d /opt/solr/core-x -c core-x -shards 1 -replicationFactor 1 -p 8983 && wait" ]
healthcheck:
<<: *healthcheck-defaults
test: [ "CMD-SHELL", "curl -s -u admin:solrRocks 'http://solr-lss-dev-solr9:8983/solr/admin/collections?action=CLUSTERSTATUS&collection=core-x' | grep -q '\"state\":\"active\"'" ]
interval: 30s
start_period: 30s
timeout: 30s
retries: 10
zoo1: # Zookeeper run in standalone mode for development and testing purposes. In production, we would use a Zookeeper ensemble for high availability.
image: zookeeper:3.9.0
container_name: zoo1
restart: always
hostname: zoo1
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=zoo1:2888:3888;2181
ZOO_4LW_COMMANDS_WHITELIST: mntr,conf,ruok
volumes:
- zoo1_data:/data
healthcheck:
<<: *healthcheck-defaults
test: [ "CMD", "nc", "-z", "-w", "2", "localhost", "2181" ] # Check if ZooKeeper is listening on port 2181
interval: 30s
start_period: 30s
timeout: 30s
retries: 10
solr-ptsearch:
# This is a core service
build: ../../ptsearch-solr #TODO: Build from image in ghcr.io
Expand All @@ -35,3 +73,6 @@ services:
healthcheck:
<<: *healthcheck-defaults
test: "curl -f --retry-connrefused http://solr-ptsearch:8983/solr/ptsearch/admin/ping"
volumes:
solr1_data: null
zoo1_data: null
21 changes: 20 additions & 1 deletion docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ services:
- ${BABEL_HOME}/catalog:/app
working_dir: /app
command: composer install

# This is Solr 6
solr-lss-dev:
extends:
file: common/docker/solr.yml
Expand All @@ -107,6 +107,23 @@ services:
- backend
- stage_item

zoo1:
extends:
file: common/docker/solr.yml
service: zoo1
profiles:
- solr9

solr-lss-dev-solr9:
extends:
file: common/docker/solr.yml
service: solr-lss-dev-solr-solr9
profiles:
- solr9
depends_on:
zoo1:
condition: service_healthy

solr-ptsearch:
extends:
file: common/docker/solr.yml
Expand Down Expand Up @@ -155,3 +172,5 @@ services:

volumes:
gem_cache:
solr1_data:
zoo1_data:
7 changes: 4 additions & 3 deletions ls/lib/Config/docker-local.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
num_shards_list = 1
mbooks_solr_engines = http://solr-lss-dev:8983/solr/core-x
A_mbooks_solr_engines = http://solr-lss-dev:8983/solr/core-x
B_mbooks_solr_engines = http://solr-lss-dev:8983/solr/core-y
mbooks_solr_engines = ENV[LSS_SOLR_ENGINE_URL_A]
A_mbooks_solr_engines = ENV[LSS_SOLR_ENGINE_URL_A]
B_mbooks_solr_engines = ENV[LSS_SOLR_ENGINE_URL_B]
solr_basic_auth_token = ENV[LSS_SOLR_BASIC_AUTH]
# debug_enabled = 1
2 changes: 1 addition & 1 deletion ls/lib/LS/Searcher/Facets.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ sub __Solr_result {
my ($C, $query_string, $rs, $AB) = @_;

my $url = $self->__get_Solr_select_url($C, $query_string, $AB);
my $req = $self->__get_request_object($url);
my $req = $self->__get_request_object($url, $C);

@moseshll moseshll Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the mdp-config with Solr auth token can be retrieved.

my $ua = $self->__create_user_agent();

if (DEBUG('query')) {
Expand Down
2 changes: 1 addition & 1 deletion mdp-lib/Institutions.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ sub get_idp_list {
$seen{$$hash_ref{inst_id}} = 1;

my $host = $ENV{'HTTP_HOST'} || 'localhost';
my $idp_url = $hash_ref->{template};
my $idp_url = $hash_ref->{template} // '';

@moseshll moseshll Aug 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will help declutter the logs, because I am certain this is one of the sites of "uninitialized value" warnings cited in ETT-1776. This is the proper mitigation.

@aelkiss aelkiss Aug 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moseshll I thought we had determined a long time ago that template wasn't actually being used, and so we haven't been keeping it up to date or even populated at all for some institutions. Is this code running but the result is never used anywhere?

$idp_url =~ s,___HOST___,$host,;
$idp_url =~ s,&amp;,&,;

Expand Down