Skip to content

Check the EIA response before saving it as a workbook - #341

Open
arpitjain099 wants to merge 1 commit into
NETL-RIC:masterfrom
arpitjain099:fix/eia7a-download-status-check
Open

arpitjain099 wants to merge 1 commit into
NETL-RIC:masterfrom
arpitjain099:fix/eia7a-download-status-check

Conversation

@arpitjain099

Copy link
Copy Markdown

eia_7a_download writes requests.get(url).content straight to coalpublic<year>.xls without looking at the status. When EIA answers 404, the error page gets saved as the workbook, and read_eia7a later raises the "see issues/230" ValueError. So a plain missing-file response is reported to the user as the XML-format problem from that issue, which is a different thing entirely.

The folder was also created before the request, so after a failed download the caller's if not os.path.exists(expected_7a_folder) guard stops trying and find_file_in_folder fails on an empty directory instead.

Now it calls raise_for_status() first, only creates the folder and writes once the response is good, and the log line names the actual error.

I checked both paths with requests.get patched: a 404 used to leave a 75-byte coalpublic2016.xls starting <html><head><title>4, and now leaves nothing and logs "404 Not Found for url: ...". A 200 still writes the file as before.

The except is now except Exception as e rather than bare, and the message is a warning rather than info, since a failed download does change what the rest of the run can do. Happy to drop that part if you would rather keep it quiet.

Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
@m-jamieson
m-jamieson requested a review from dt-woods October 2, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant