diff --git a/Gemfile b/Gemfile index 221286d..3889803 100644 --- a/Gemfile +++ b/Gemfile @@ -8,6 +8,8 @@ group :development, :test do gem "minitest" gem "railties" # test/rails boots an application on it gem "rake" + gem "rspec-core" # test/testing_test.rb runs Gamba's matchers in RSpec + gem "rspec-expectations" gem "rubocop-performance", require: false gem "standard", require: false gem "vernier" diff --git a/README.md b/README.md index 8fab19e..953a5e0 100644 --- a/README.md +++ b/README.md @@ -192,15 +192,38 @@ To report a vulnerability, see [SECURITY.md](SECURITY.md). ## Checking your app's PDFs +Tests that check a PDF's bytes break on every harmless change. Gamba's own checks, those its oracle compares it with +Prawn by, come as assertions for Minitest and matchers for RSpec, loaded only on request. They need `mutool`, and +`pdftoppm` and `qpdf` for pages and validity; a missing tool fails with its install command, or skips the check +with `Gamba::Testing.missing_tools = :skip`. + +```ruby +require "gamba/minitest" # in test_helper.rb + +assert_pdf_text pdf, "Invoice INV-2026-0001", page: 1 # bytes or a path; a Regexp works too +assert_same_pages expected_pdf, pdf # text, positions and pixels in MuPDF and Poppler +assert_valid_pdf pdf # qpdf and Poppler +assert_pdf_a pdf # veraPDF +``` + +```ruby +require "gamba/rspec" # in spec_helper.rb + +expect(pdf).to have_pdf_text("Invoice INV-2026-0001", page: 1) +expect(pdf).to have_same_pages_as(expected_pdf) +expect(pdf).to be_a_valid_pdf +``` + Before switching, you can compare what your app produces under both libraries. `script/capture_pdfs.rb` saves -every PDF your test suite renders, named after the test. Run the suite once with Prawn and once with Gamba: +every PDF your test suite renders, named after the test. Run the suite once with Prawn and once with Gamba, then +compare the two directories with the same checks: ```sh GAMBA_CAPTURE_DIRECTORY=tmp/prawn RUBYOPT=-r/path/to/gamba/script/capture_pdfs.rb bin/rails test +GAMBA_CAPTURE_DIRECTORY=tmp/gamba RUBYOPT=-r/path/to/gamba/script/capture_pdfs.rb bin/rails test +ruby /path/to/gamba/script/capture_pdfs.rb tmp/prawn tmp/gamba ``` -Then compare the two directories with `Oracle.pdf_differences`, from Gamba's test support. - ## Performance `bundle exec rake bench` renders each document in `bench/documents` with Prawn 2.5 and with Gamba, in 7 fresh diff --git a/docs/guide.html b/docs/guide.html index c54a427..abee7b6 100644 --- a/docs/guide.html +++ b/docs/guide.html @@ -307,8 +307,16 @@

Servers that fork

Comparing an app’s PDFs

-

script/capture_pdfs.rb writes every PDF an app’s test suite renders, named after its test. Run the suite under Prawn, then under Gamba, and compare both directories with Oracle.pdf_differences:

-
GAMBA_CAPTURE_DIRECTORY=tmp/prawn RUBYOPT=-r/path/to/gamba/script/capture_pdfs.rb bin/rails test
+

The checks Gamba’s oracle compares it with Prawn by come as assertions for Minitest and matchers for RSpec, loaded only on request. A PDF is given as its bytes or a file’s path. A tool that is missing fails with its install command, or skips the check with Gamba::Testing.missing_tools = :skip; a check never passes unchecked.

+
require "gamba/minitest" # or "gamba/rspec": have_pdf_text, have_same_pages_as, be_a_valid_pdf, be_pdf_a
+
+assert_pdf_text pdf, "Invoice INV-2026-0001", page: 1, at: [36, 52] # at: a baseline, from the top left
+assert_same_pages expected_pdf, pdf
+assert_valid_pdf pdf
+assert_pdf_a pdf
+

A failure says what the page reads instead, or names the page and the glyph that moved, or the pixel rows that differ in MuPDF or Poppler. script/capture_pdfs.rb writes every PDF an app’s test suite renders, named after its test. Run the suite under Prawn, then under Gamba, and compare both directories with the same checks:

+
GAMBA_CAPTURE_DIRECTORY=tmp/prawn RUBYOPT=-r/path/to/gamba/script/capture_pdfs.rb bin/rails test
+ruby /path/to/gamba/script/capture_pdfs.rb tmp/prawn tmp/gamba
diff --git a/gemfiles/rails_8_0.gemfile b/gemfiles/rails_8_0.gemfile index 25150df..e06acbc 100644 --- a/gemfiles/rails_8_0.gemfile +++ b/gemfiles/rails_8_0.gemfile @@ -7,6 +7,8 @@ gemspec path: ".." gem "minitest" gem "railties", "~> 8.0.0" gem "rake" +gem "rspec-core" +gem "rspec-expectations" gem "rubocop-performance", require: false gem "standard", require: false gem "vernier" diff --git a/lib/gamba/minitest.rb b/lib/gamba/minitest.rb new file mode 100644 index 0000000..6614e23 --- /dev/null +++ b/lib/gamba/minitest.rb @@ -0,0 +1,35 @@ +# frozen_string_literal: true + +require "gamba/testing" + +module Gamba + # require "gamba/minitest": assertions on PDFs, as bytes or a file's path, in every Minitest::Test. + module Minitest + def assert_pdf_text(pdf, text, page: nil, at: nil) + gamba_check { Gamba::Testing.text_failure(pdf, text, page:, at:) } + end + + def assert_same_pages(expected, actual) + gamba_check { Gamba::Testing.pages_failure(expected, actual) } + end + + def assert_valid_pdf(pdf) + gamba_check { Gamba::Testing.validity_failure(pdf) } + end + + def assert_pdf_a(pdf, flavour: "3b") + gamba_check { Gamba::Testing.pdf_a_failure(pdf, flavour:) } + end + + private + + def gamba_check + failure = yield + assert failure.nil?, failure + rescue Gamba::PdfCheck::ToolMissing => error + (Gamba::Testing.missing_tools == :skip) ? skip(error.message) : flunk(error.message) + end + end +end + +Minitest::Test.include(Gamba::Minitest) if defined?(Minitest::Test) diff --git a/lib/gamba/pdf_check.rb b/lib/gamba/pdf_check.rb new file mode 100644 index 0000000..eeb582d --- /dev/null +++ b/lib/gamba/pdf_check.rb @@ -0,0 +1,220 @@ +# frozen_string_literal: true + +require "open3" +require "pathname" +require "tempfile" +require "tmpdir" + +module Gamba + # What tells two PDFs apart, or a PDF from a valid one, through tools that share no code with Gamba: MuPDF for the + # text and its positions, MuPDF and Poppler for pixels, qpdf and Poppler for validity, veraPDF for PDF/A. Gamba's + # own oracle compares it with Prawn this way; Gamba::Minitest and Gamba::RSpec put it in an application's tests. + # A PDF is given as its bytes or as the path of a file. + module PdfCheck + # Each glyph: [character, font without its subset tag, size, x, y], positions in points from the top left. + Page = Data.define(:size, :glyphs) + + RESOLUTION_DPI = 150 + POSITION_PRECISION_DIGITS = 2 + # Same character, font and size, at most this far apart: positions round to 0.01 on either side of a boundary. + POSITION_TOLERANCE = 0.0101 + RENDERERS = {"MuPDF" => :rasterize, "Poppler" => :rasterize_with_poppler}.freeze + QPDF_WARNINGS_EXIT_STATUS = 3 + TOOLS = { + "mutool" => "brew install mupdf-tools, or apt-get install mupdf-tools", + "pdftoppm" => "brew install poppler, or apt-get install poppler-utils", + "pdftotext" => "brew install poppler, or apt-get install poppler-utils", + "qpdf" => "brew install qpdf, or apt-get install qpdf", + "verapdf" => "brew install verapdf, or https://verapdf.org/software/" + }.freeze + + # Raised when a check needs a tool that is not installed: the message names the install command. + class ToolMissing < StandardError; end + + class << self + # A password encrypted PDFs open with: the tools are given it when they cannot open one without. + attr_accessor :password + + # The pages of the PDF, their text read by MuPDF. + def text_layout(pdf) + with_path(pdf) { |path| parse_text(mutool("draw", "-F", "stext", "-o", "-", path)) } + end + + # The text of each page by line, top to bottom: what a reader sees, words joined. + def text_lines(pdf) + text_layout(pdf).map { |page| page.glyphs.group_by(&:last).sort.map! { |_y, glyphs| glyphs.map(&:first).join } } + end + + # What makes qpdf or Poppler find the PDF broken, nil when nothing does. qpdf exits 3 for warnings: a PDF readers + # have to repair is not a valid one. Poppler checks the operands of content stream operators, which qpdf does not. + def invalidity(pdf) + with_path(pdf) do |path| + output, status = run("qpdf", "--check", path) + output, status = run("qpdf", "--check", "--password=#{password}", path) if password && output.include?("invalid password") + next output.lines.grep(/WARNING|ERROR|error/).first(3).join.strip.then { |problem| problem.empty? ? "qpdf exit #{status.exitstatus}" : problem } unless status.success? + + errors, _status = run("pdftotext", *(password ? ["-upw", password] : []), path, File::NULL) + errors.lines.grep(/Syntax Error/).first(3).join.strip.then { |problem| problem unless problem.empty? } + end + end + + # What veraPDF finds against the PDF/A flavour, nil when it passes. + def pdf_a_failure(pdf, flavour: "3b") + with_path(pdf) do |path| + report, _status = run("verapdf", "--flavour", flavour, "--format", "text", path) + report.start_with?("PASS") ? nil : report.strip + end + end + + # Every difference between the pages of two PDFs: their number and sizes, each glyph with its font and position, + # then pixel rows in MuPDF and in Poppler. Empty when a reader could not tell them apart. + def page_differences(expected, actual) + with_path(expected) do |expected_path| + with_path(actual) do |actual_path| + text_differences(expected_path, actual_path) + pixel_differences(expected_path, actual_path) + end + end + end + + def text_differences(expected, actual) + expected_pages = text_layout(expected) + actual_pages = text_layout(actual) + return ["#{actual_pages.size} pages instead of #{expected_pages.size}"] if expected_pages.size != actual_pages.size + + expected_pages.zip(actual_pages).each_with_index.filter_map do |(expected_page, actual_page), index| + next if expected_page.size == actual_page.size && expected_page.glyphs.size == actual_page.glyphs.size && + expected_page.glyphs.zip(actual_page.glyphs).all? { |expected_glyph, actual_glyph| same_glyph?(expected_glyph, actual_glyph) } + next "page #{index + 1}: size #{actual_page.size} instead of #{expected_page.size}" if expected_page.size != actual_page.size + + position = expected_page.glyphs.zip(actual_page.glyphs).index { |expected_glyph, actual_glyph| !same_glyph?(expected_glyph, actual_glyph) } + next "page #{index + 1}: #{actual_page.glyphs.size} glyphs instead of #{expected_page.glyphs.size}" unless position + + "page #{index + 1}: glyph #{position} is #{actual_page.glyphs[position].inspect} instead of #{expected_page.glyphs[position].inspect}" + end + end + + def same_glyph?(expected_glyph, actual_glyph) + return false if actual_glyph.nil? || expected_glyph.first(3) != actual_glyph.first(3) + + expected_glyph.last(2).zip(actual_glyph.last(2)).all? { |expected, actual| (expected - actual).abs <= POSITION_TOLERANCE } + end + + # Two renderers that share no code, MuPDF and Poppler, must each draw both documents identically. Subset tags + # are made the same first: MuPDF hints glyphs differently whether a tag is valid or not. + def pixel_differences(expected, actual) + with_path(expected) do |expected_path| + with_path(actual) do |actual_path| + Dir.mktmpdir do |directory| + expected_path = with_neutral_subset_tags(expected_path, File.join(directory, "expected.pdf")) + actual_path = with_neutral_subset_tags(actual_path, File.join(directory, "actual.pdf")) + rendered_pixel_differences(expected_path, actual_path) + end + end + end + end + + def rendered_pixel_differences(expected, actual) + RENDERERS.flat_map do |renderer, rasterizer| + Dir.mktmpdir do |directory| + expected_images = send(rasterizer, expected, File.join(directory, "expected")) + actual_images = send(rasterizer, actual, File.join(directory, "actual")) + expected_images.zip(actual_images).each_with_index.filter_map do |(expected_image, actual_image), index| + next if actual_image.nil? || File.binread(expected_image) == File.binread(actual_image) + + "page #{index + 1}: #{differing_rows(expected_image, actual_image)} pixel rows differ at #{RESOLUTION_DPI} dpi#{" in #{renderer}" unless renderer == "MuPDF"}" + end + end + end + end + + # Each page as an image file, by MuPDF. + def rasterize(pdf, prefix) + mutool("draw", "-r", RESOLUTION_DPI.to_s, "-o", "#{prefix}-%04d.pnm", pdf) + Dir["#{prefix}-*.pnm"].sort + end + + def rasterize_with_poppler(pdf, prefix) + _output, status = run("pdftoppm", "-r", RESOLUTION_DPI.to_s, pdf, prefix) + run("pdftoppm", "-r", RESOLUTION_DPI.to_s, "-upw", password, pdf, prefix) if password && !status.success? + Dir["#{prefix}-*.ppm"].sort + end + + def mutool(*arguments, path) + output, errors, status = Open3.capture3(tool("mutool"), arguments.first, "-q", *password_arguments("-p"), *arguments.drop(1), path, binmode: true) + raise "mutool #{arguments.first} failed on #{path}: #{errors}" unless status.success? + + output + end + + # The bytes of a PDF written to a file for the tools, or the path given (a String or a Pathname). + def with_path(pdf) + return yield pdf.to_s if pdf.is_a?(Pathname) || (!pdf.b.start_with?("%PDF") && !pdf.include?("\0") && File.file?(pdf)) + + Tempfile.create(["gamba-check", ".pdf"]) do |file| + file.binmode + file.write(pdf) + file.flush + yield file.path + end + end + + # The PDF written to path with every subset tag the same. Only names after /BaseFont or /FontName change: compressed stream data can hold a tag's bytes by chance. + def with_neutral_subset_tags(pdf, path) + _output, status = run("qpdf", "--qdf", "--object-streams=disable", "--stream-data=preserve", pdf, path) + return pdf unless status.success? || status.exitstatus == QPDF_WARNINGS_EXIT_STATUS + + File.binwrite(path, File.binread(path).gsub(%r{(/(?:BaseFont|FontName)\s*)/[0-9A-Za-z]{6}\+}n, '\1/AAAAAA+')) + path + end + + private + + def parse_text(xml) + xml.split("]*)>/) do |element, attributes| + value_by_name = attributes.scan(/(\w+)="([^"]*)"/).to_h + if element == "font" + font = [value_by_name.fetch("name").sub(/\A[A-Z]{6}\+/, ""), value_by_name.fetch("size").to_f.round(POSITION_PRECISION_DIGITS)] + else + position = value_by_name.fetch_values("x", "y").map { _1.to_f.round(POSITION_PRECISION_DIGITS) } + glyphs << [value_by_name.fetch("c"), *font, *position] + end + end + Page.new(size: page[/width="[^"]*" height="[^"]*"/], glyphs:) + end + end + + def differing_rows(expected_image, actual_image) + expected_rows = pnm_rows(expected_image) + actual_rows = pnm_rows(actual_image) + return "all" if expected_rows.size != actual_rows.size + + expected_rows.zip(actual_rows).count { |expected_row, actual_row| expected_row != actual_row } + end + + def pnm_rows(image) + content = File.binread(image) + header = content[/\AP\d\s+\d+\s+\d+\s+\d+\s/] + _format, width, height = header.split.map { Integer(_1[/\d+/]) } + row_bytes = width * (content.start_with?("P5") ? 1 : 3) + Array.new(height) { |row| content.byteslice(header.bytesize + (row * row_bytes), row_bytes) } + end + + def password_arguments(flag) + password ? [flag, password] : [] + end + + def run(name, *arguments) + Open3.capture2e(tool(name), *arguments) + end + + def tool(name) + @found ||= {} + @found[name] ||= ENV.fetch("PATH", "").split(File::PATH_SEPARATOR).map { File.join(_1, name) }.find { File.executable?(_1) } || + raise(ToolMissing, "#{name} is not installed: #{TOOLS.fetch(name)}") + end + end + end +end diff --git a/lib/gamba/rspec.rb b/lib/gamba/rspec.rb new file mode 100644 index 0000000..3f6feaa --- /dev/null +++ b/lib/gamba/rspec.rb @@ -0,0 +1,58 @@ +# frozen_string_literal: true + +require "gamba/testing" + +module Gamba + # require "gamba/rspec": matchers on PDFs, as bytes or a file's path, in every example group. + module RSpec + # A matcher whose check returns nil when it passes, or the failure's words. + class Matcher + def initialize(description, &check) + @description = description + @check = check + end + + def matches?(pdf) + @failure = @check.call(pdf) + @failure.nil? + rescue Gamba::PdfCheck::ToolMissing => error + raise if Gamba::Testing.missing_tools == :fail + + ::RSpec::Core::Pending.mark_skipped!(::RSpec.current_example, error.message) if defined?(::RSpec::Core::Pending) + raise ::RSpec::Core::Pending::SkipDeclaredInExample, error.message + end + + def does_not_match?(pdf) + !matches?(pdf) + end + + def failure_message + @failure + end + + def failure_message_when_negated + "expected the PDF not to #{@description}" + end + + attr_reader :description + end + + def have_pdf_text(text, page: nil, at: nil) + Matcher.new("have the text #{text.inspect}") { |pdf| Gamba::Testing.text_failure(pdf, text, page:, at:) } + end + + def have_same_pages_as(expected) + Matcher.new("have the same pages") { |actual| Gamba::Testing.pages_failure(expected, actual) } + end + + def be_a_valid_pdf + Matcher.new("be a valid PDF") { |pdf| Gamba::Testing.validity_failure(pdf) } + end + + def be_pdf_a(flavour: "3b") + Matcher.new("be PDF/A-#{flavour}") { |pdf| Gamba::Testing.pdf_a_failure(pdf, flavour:) } + end + end +end + +RSpec.configure { |config| config.include(Gamba::RSpec) } if defined?(RSpec.configure) diff --git a/lib/gamba/testing.rb b/lib/gamba/testing.rb new file mode 100644 index 0000000..15b8d81 --- /dev/null +++ b/lib/gamba/testing.rb @@ -0,0 +1,79 @@ +# frozen_string_literal: true + +require_relative "pdf_check" + +module Gamba + # What an application's tests ask of its PDFs, in words of the test frameworks: Gamba::Minitest and Gamba::RSpec + # put it in their own forms. A check whose tool is missing fails with the install command, or is skipped when + # Gamba::Testing.missing_tools is :skip; it never passes unchecked. + module Testing + MISSING_TOOLS = %i[fail skip].freeze + # Text found this many points from where it was expected is where it was expected. + POSITION_TOLERANCE = 1 + + class << self + attr_reader :missing_tools + + def missing_tools=(choice) + raise ArgumentError, "missing_tools is #{choice.inspect}: give :fail or :skip" unless MISSING_TOOLS.include?(choice) + + @missing_tools = choice + end + + # nil when the text is on the page (any page unless told), else what was there instead. text: a String, whose + # spaces match any run of them, or a Regexp; at: [x, y] of its first character's baseline, in points from the page's + # top left, as MuPDF reads it. + def text_failure(pdf, text, page: nil, at: nil) + pages = PdfCheck.text_layout(pdf) + candidates = page ? [pages[page - 1]].compact : pages + return "the PDF has #{pages.size} pages, not #{page}" if candidates.empty? + + pattern = text.is_a?(Regexp) ? text : Regexp.new(text.split.map { Regexp.escape(_1) }.join("\\s+")) + found = candidates.any? { |candidate| lines(candidate).any? { |glyphs| occurs?(glyphs, pattern, at) } } + return nil if found + + place = page ? "page #{page}" : "the PDF" + "#{text.inspect} is not on #{place}#{" at #{at.inspect}" if at}: it reads\n#{candidates.map { |candidate| lines(candidate).map { _1.map(&:first).join }.join("\n") }.join("\n---\n")}" + end + + def pages_failure(expected, actual) + differences = PdfCheck.page_differences(expected, actual) + differences.empty? ? nil : "the pages differ:\n#{differences.join("\n")}" + end + + def validity_failure(pdf) + problem = PdfCheck.invalidity(pdf) + problem && "the PDF is not valid: #{problem}" + end + + def pdf_a_failure(pdf, flavour: "3b") + report = PdfCheck.pdf_a_failure(pdf, flavour:) + report && "the PDF is not PDF/A-#{flavour}:\n#{report}" + end + + private + + def lines(page) + page.glyphs.group_by(&:last).sort.map!(&:last) + end + + # Whether the pattern matches the line, from a glyph near `at` when told. + def occurs?(glyphs, pattern, at) + line = glyphs.map(&:first).join + position = 0 + while (match = pattern.match(line, position)) + return true if at.nil? || near?(glyphs[match.begin(0)], at) + + position = match.begin(0) + 1 + end + false + end + + def near?(glyph, (x, y)) + glyph && (glyph[3] - x).abs <= POSITION_TOLERANCE && (glyph[4] - y).abs <= POSITION_TOLERANCE + end + end + + self.missing_tools = :fail + end +end diff --git a/script/capture_pdfs.rb b/script/capture_pdfs.rb index 85aa4f7..9204dba 100644 --- a/script/capture_pdfs.rb +++ b/script/capture_pdfs.rb @@ -2,8 +2,31 @@ # Loaded with RUBYOPT=-r…/capture_pdfs.rb into an app's test run: every PDF a test renders is written to # GAMBA_CAPTURE_DIRECTORY, named after the test, so the same run under Prawn and under Gamba can be compared. +# Run as `ruby capture_pdfs.rb tmp/prawn tmp/gamba`, it compares them with the checks of gamba/minitest and +# gamba/rspec, and exits 1 when a PDF differs. require "fileutils" +if $PROGRAM_NAME == __FILE__ + require_relative "../lib/gamba/pdf_check" + + expected_directory, actual_directory = ARGV + abort "usage: ruby #{$PROGRAM_NAME} PRAWN_DIRECTORY GAMBA_DIRECTORY" unless actual_directory + + names = (Dir.children(expected_directory) | Dir.children(actual_directory)).grep(/\.pdf\z/).sort + differing = names.count do |name| + expected = File.join(expected_directory, name) + actual = File.join(actual_directory, name) + differences = if !File.exist?(actual) then ["missing from #{actual_directory}"] + elsif !File.exist?(expected) then ["missing from #{expected_directory}"] + else Gamba::PdfCheck.page_differences(expected, actual) + end + puts "#{name}: #{differences.join("; ")}" unless differences.empty? + !differences.empty? + end + puts "#{names.size} PDFs, #{differing} differing" + exit(differing.zero? ? 0 : 1) +end + module GambaCapture DIRECTORY = ENV.fetch("GAMBA_CAPTURE_DIRECTORY") @count_by_test = Hash.new(0) diff --git a/test/support/oracle.rb b/test/support/oracle.rb index 29c4969..f5fdd4b 100644 --- a/test/support/oracle.rb +++ b/test/support/oracle.rb @@ -4,6 +4,7 @@ require "fileutils" require "open3" require "tmpdir" +require_relative "../../lib/gamba/pdf_check" # Renders the same examples with Prawn and with Gamba, each in its own process, and lists what tells their PDFs # apart: pages, text and its positions, pixels, validity, weight. @@ -11,10 +12,9 @@ module Oracle ROOT = File.expand_path("../..", __dir__) MANUAL_DIRECTORY = File.join(ROOT, "corpus/cache/prawn/manual") RENDER_SCRIPT = File.join(__dir__, "oracle_render.rb") - RESOLUTION_DPI = 150 - POSITION_PRECISION_DIGITS = 2 - # Every encrypted example of Prawn's security chapter uses this user password; mutool and qpdf ignore it elsewhere. + # Every encrypted example of Prawn's security chapter uses this user password; the tools ignore it elsewhere. MANUAL_USER_PASSWORD = "foo" + Gamba::PdfCheck.password = MANUAL_USER_PASSWORD ENGINES = { "prawn" => {gemfile: File.join(ROOT, "oracle/Gemfile"), load_path: []}, @@ -108,134 +108,37 @@ def self.error_class_differences(reference_error, candidate_error) (reference_class == candidate_class) ? [] : ["raises #{candidate_class} instead of #{reference_class}"] end - # qpdf exits with 3 for warnings: a PDF readers have to repair is not a valid one. qpdf does not check the operands - # of content stream operators: Poppler does, reporting a syntax error. + # The comparisons themselves are Gamba::PdfCheck's, which applications' tests use too: one implementation. def self.invalidity(pdf) - output, status = Open3.capture2e("qpdf", "--check", pdf) - output, status = Open3.capture2e("qpdf", "--check", "--password=#{MANUAL_USER_PASSWORD}", pdf) if output.include?("invalid password") - return output.lines.grep(/WARNING|ERROR|error/).first(3).join.strip.then { |problem| problem.empty? ? "qpdf exit #{status.exitstatus}" : problem } unless status.success? - - errors, _status = Open3.capture2e("pdftotext", "-upw", MANUAL_USER_PASSWORD, pdf, File::NULL) - errors.lines.grep(/Syntax Error/).first(3).join.strip.then { |problem| problem unless problem.empty? } + Gamba::PdfCheck.invalidity(pdf) end def self.text_differences(reference_pdf, candidate_pdf) - reference_pages = text_layout(reference_pdf) - candidate_pages = text_layout(candidate_pdf) - return ["#{candidate_pages.size} pages instead of #{reference_pages.size}"] if reference_pages.size != candidate_pages.size - - reference_pages.zip(candidate_pages).each_with_index.filter_map do |(reference_page, candidate_page), index| - next if reference_page.size == candidate_page.size && reference_page.glyphs.size == candidate_page.glyphs.size && - reference_page.glyphs.zip(candidate_page.glyphs).all? { |reference_glyph, candidate_glyph| same_glyph?(reference_glyph, candidate_glyph) } - next "page #{index + 1}: size #{candidate_page.size} instead of #{reference_page.size}" if reference_page.size != candidate_page.size - - position = reference_page.glyphs.zip(candidate_page.glyphs).index { |reference_glyph, candidate_glyph| !same_glyph?(reference_glyph, candidate_glyph) } - next "page #{index + 1}: #{candidate_page.glyphs.size} glyphs instead of #{reference_page.glyphs.size}" unless position - - "page #{index + 1}: glyph #{position} is #{candidate_page.glyphs[position].inspect} instead of #{reference_page.glyphs[position].inspect}" - end - end - - POSITION_TOLERANCE = 0.0101 - - # Same character, font and size, at most 0.01 pt apart: positions round to 0.01 on either side of a boundary. - def self.same_glyph?(reference_glyph, candidate_glyph) - return false if candidate_glyph.nil? || reference_glyph.first(3) != candidate_glyph.first(3) - - reference_glyph.last(2).zip(candidate_glyph.last(2)).all? { |reference, candidate| (reference - candidate).abs <= POSITION_TOLERANCE } + Gamba::PdfCheck.text_differences(reference_pdf, candidate_pdf) end - # Each glyph is [character, font without its subset tag, font size, x, y]. - Page = Data.define(:size, :glyphs) - def self.text_layout(pdf) - xml = mutool("draw", "-F", "stext", "-o", "-", pdf) - xml.split("]*)>/) do |element, attributes| - value_by_name = attributes.scan(/(\w+)="([^"]*)"/).to_h - if element == "font" - font = [value_by_name.fetch("name").sub(/\A[A-Z]{6}\+/, ""), value_by_name.fetch("size").to_f.round(POSITION_PRECISION_DIGITS)] - else - position = value_by_name.fetch_values("x", "y").map { _1.to_f.round(POSITION_PRECISION_DIGITS) } - glyphs << [value_by_name.fetch("c"), *font, *position] - end - end - Page.new(size: page[/width="[^"]*" height="[^"]*"/], glyphs:) - end + Gamba::PdfCheck.text_layout(pdf) end - # Two renderers that share no code, MuPDF and Poppler, must each draw both documents identically. - RENDERERS = {"MuPDF" => :rasterize, "Poppler" => :rasterize_with_poppler}.freeze - QPDF_WARNINGS_EXIT_STATUS = 3 - def self.pixel_differences(reference_pdf, candidate_pdf) - Dir.mktmpdir do |tagged_directory| - reference_pdf = with_neutral_subset_tags(reference_pdf, File.join(tagged_directory, "reference.pdf")) - candidate_pdf = with_neutral_subset_tags(candidate_pdf, File.join(tagged_directory, "candidate.pdf")) - rendered_pixel_differences(reference_pdf, candidate_pdf) - end + Gamba::PdfCheck.pixel_differences(reference_pdf, candidate_pdf) end def self.rendered_pixel_differences(reference_pdf, candidate_pdf) - RENDERERS.flat_map do |renderer, rasterizer| - Dir.mktmpdir do |directory| - reference_images = send(rasterizer, reference_pdf, File.join(directory, "reference")) - candidate_images = send(rasterizer, candidate_pdf, File.join(directory, "candidate")) - reference_images.zip(candidate_images).each_with_index.filter_map do |(reference_image, candidate_image), index| - next if candidate_image.nil? || File.binread(reference_image) == File.binread(candidate_image) - - "page #{index + 1}: #{differing_rows(reference_image, candidate_image)} pixel rows differ at #{RESOLUTION_DPI} dpi#{" in #{renderer}" unless renderer == "MuPDF"}" - end - end - end + Gamba::PdfCheck.rendered_pixel_differences(reference_pdf, candidate_pdf) end - # MuPDF hints some glyphs differently whether a font's subset tag is valid or not, and Prawn's lowercase hexadecimal - # tags are not ("exactly six uppercase letters", ISO 32000-1 §9.6.4): both documents get the same tag before drawing. - # Only names after /BaseFont or /FontName change: compressed stream data can hold a tag's bytes by chance, and - # rewriting them there would corrupt that stream. def self.with_neutral_subset_tags(pdf, path) - _output, status = Open3.capture2e("qpdf", "--qdf", "--object-streams=disable", "--stream-data=preserve", pdf, path) - return pdf unless status.success? || status.exitstatus == QPDF_WARNINGS_EXIT_STATUS - - File.binwrite(path, File.binread(path).gsub(%r{(/(?:BaseFont|FontName)\s*)/[0-9A-Za-z]{6}\+}n, '\1/AAAAAA+')) - path - end - - def self.rasterize_with_poppler(pdf, prefix) - _output, status = Open3.capture2e("pdftoppm", "-r", RESOLUTION_DPI.to_s, pdf, prefix) - Open3.capture2e("pdftoppm", "-r", RESOLUTION_DPI.to_s, "-upw", MANUAL_USER_PASSWORD, pdf, prefix) unless status.success? - Dir["#{prefix}-*.ppm"].sort + Gamba::PdfCheck.with_neutral_subset_tags(pdf, path) end def self.rasterize(pdf, prefix) - mutool("draw", "-r", RESOLUTION_DPI.to_s, "-o", "#{prefix}-%04d.pnm", pdf) - Dir["#{prefix}-*.pnm"].sort - end - - def self.differing_rows(reference_image, candidate_image) - reference_rows = pnm_rows(reference_image) - candidate_rows = pnm_rows(candidate_image) - return "all" if reference_rows.size != candidate_rows.size - - reference_rows.zip(candidate_rows).count { |reference_row, candidate_row| reference_row != candidate_row } - end - - def self.pnm_rows(image) - content = File.binread(image) - header = content[/\AP\d\s+\d+\s+\d+\s+\d+\s/] - _format, width, height = header.split.map { Integer(_1[/\d+/]) } - row_bytes = width * (content.start_with?("P5") ? 1 : 3) - Array.new(height) { |row| content.byteslice(header.bytesize + (row * row_bytes), row_bytes) } + Gamba::PdfCheck.rasterize(pdf, prefix) end def self.mutool(*arguments, pdf) - output, errors, status = Open3.capture3("mutool", *arguments.take(1), "-q", "-p", MANUAL_USER_PASSWORD, *arguments.drop(1), pdf, binmode: true) - raise "mutool #{arguments.first} failed on #{pdf}: #{errors}" unless status.success? - - output + Gamba::PdfCheck.mutool(*arguments, pdf) end def self.read_if_present(directory, filename) diff --git a/test/testing_test.rb b/test/testing_test.rb new file mode 100644 index 0000000..e95487c --- /dev/null +++ b/test/testing_test.rb @@ -0,0 +1,133 @@ +# frozen_string_literal: true + +require_relative "test_helper" +require "prawn" +require "fileutils" +require "open3" +require "rbconfig" +require "gamba/minitest" + +# The assertions an application's tests get from require "gamba/minitest" or "gamba/rspec": the oracle's checks, +# failing in words that name the page and the glyph, and never passing when a tool is missing. +class TestingTest < Minitest::Test + INVOICE = Prawn::Document.new do + text "Invoice INV-2026-0001", size: 18 + start_new_page + text_box "Total 1 234,00", at: [100, 600] + end.render.freeze + + def teardown + Gamba::Testing.missing_tools = :fail + end + + def test_text_is_found_on_any_page_or_the_one_told_and_where_told + assert_pdf_text INVOICE, "Invoice INV-2026-0001" + assert_pdf_text INVOICE, "Total 1 234,00", page: 2 + assert_pdf_text INVOICE, /INV-\d{4}-\d{4}/, page: 1 + assert_pdf_text INVOICE, "Total", page: 2, at: [136, 164.6] + end + + def test_missing_text_fails_saying_what_the_page_reads + failure = assert_raises(Minitest::Assertion) { assert_pdf_text INVOICE, "Invoice", page: 2 } + assert_equal "\"Invoice\" is not on page 2: it reads\nTotal 1 234,00", failure.message + + assert_raises(Minitest::Assertion) { assert_pdf_text INVOICE, "Total", page: 2, at: [36, 164.6] } + assert_match(/\Athe PDF has 2 pages, not 3/, assert_raises(Minitest::Assertion) { assert_pdf_text INVOICE, "Total", page: 3 }.message) + end + + def test_pages_are_the_same_or_their_differences_are_named + assert_same_pages INVOICE, INVOICE.dup + moved = Prawn::Document.new do + text "Invoice INV-2026-0001", size: 18 + start_new_page + text_box "Total 1 234,00", at: [101, 600] + end.render + + failure = assert_raises(Minitest::Assertion) { assert_same_pages INVOICE, moved } + assert_match(/\Athe pages differ:\npage 2: glyph 0 is \["T", "Helvetica", 12.0, 137.0, [\d.]+\] instead of \["T", "Helvetica", 12.0, 136.0, [\d.]+\]$/, failure.message) + assert_includes failure.message, "pixel rows differ at 150 dpi in Poppler" + end + + def test_validity_is_checked_by_qpdf_and_poppler + assert_valid_pdf INVOICE + broken = INVOICE.sub(/startxref\s+\d+/, "startxref\n1") + + assert_match(/\Athe PDF is not valid: /, assert_raises(Minitest::Assertion) { assert_valid_pdf broken }.message) + end + + def test_paths_work_as_well_as_bytes + Tempfile.create(["invoice", ".pdf"]) do |file| + file.binmode + file.write(INVOICE) + file.flush + + assert_pdf_text file.path, "Invoice" + assert_same_pages Pathname.new(file.path), INVOICE + end + end + + def test_a_missing_tool_fails_with_its_install_command_or_skips_when_told + without_tools do + failure = assert_raises(Minitest::Assertion) { assert_valid_pdf INVOICE } + assert_equal "qpdf is not installed: brew install qpdf, or apt-get install qpdf", failure.message + + Gamba::Testing.missing_tools = :skip + assert_raises(Minitest::Skip) { assert_pdf_text INVOICE, "Invoice" } + end + assert_raises(ArgumentError) { Gamba::Testing.missing_tools = :pass } + end + + def test_the_rspec_matchers_pass_fail_and_skip_as_the_assertions_do + spec = <<~RUBY + require "prawn" + require "gamba/rspec" + PDF = Prawn::Document.new { text "Invoice INV-2026-0001" }.render + RSpec.describe "an invoice" do + it("has its number") { expect(PDF).to have_pdf_text("INV-2026-0001", page: 1) } + it("is valid") { expect(PDF).to be_a_valid_pdf } + it("has the same pages") { expect(PDF).to have_same_pages_as(PDF.dup) } + it("lacks a total") { expect(PDF).not_to have_pdf_text("Total") } + it("fails with what the page reads") { expect(PDF).to have_pdf_text("Total") } + it("skips when told and qpdf is missing") do + Gamba::Testing.missing_tools = :skip + Gamba::PdfCheck.instance_variable_set(:@found, nil) + ENV["PATH"] = "" + expect(PDF).to be_a_valid_pdf + end + end + RUBY + output = Dir.mktmpdir do |directory| + File.write(File.join(directory, "invoice_spec.rb"), spec) + `#{RbConfig.ruby} -I#{File.expand_path("../lib", __dir__)} -S rspec #{File.join(directory, "invoice_spec.rb")} 2>&1` + end + + assert_includes output, "6 examples, 1 failure, 1 pending" + assert_includes output, "qpdf is not installed: brew install qpdf, or apt-get install qpdf" + assert_includes output, "\"Total\" is not on the PDF: it reads\n Invoice INV-2026-0001" + end + + def test_capture_pdfs_compares_two_directories_with_the_same_checks + output, status = Dir.mktmpdir do |directory| + %w[prawn gamba].each { FileUtils.mkdir_p(File.join(directory, _1)) } + File.binwrite(File.join(directory, "prawn/same.pdf"), INVOICE) + File.binwrite(File.join(directory, "gamba/same.pdf"), INVOICE) + File.binwrite(File.join(directory, "prawn/only.pdf"), INVOICE) + Open3.capture2e(RbConfig.ruby, File.expand_path("../script/capture_pdfs.rb", __dir__), File.join(directory, "prawn"), File.join(directory, "gamba")) + end + + assert_equal 1, status.exitstatus + assert_match(%r{\Aonly.pdf: missing from .*/gamba\n2 PDFs, 1 differing\n\z}, output) + end + + private + + def without_tools + path = ENV.fetch("PATH") + ENV["PATH"] = "" + Gamba::PdfCheck.instance_variable_set(:@found, nil) + yield + ensure + ENV["PATH"] = path + Gamba::PdfCheck.instance_variable_set(:@found, nil) + end +end