{"id":2998,"date":"2025-02-01T23:04:51","date_gmt":"2025-02-01T23:04:51","guid":{"rendered":"https:\/\/timallanwheeler.com\/blog\/?p=2998"},"modified":"2025-02-01T23:04:51","modified_gmt":"2025-02-01T23:04:51","slug":"coding-your-own-tools","status":"publish","type":"post","link":"https:\/\/timallanwheeler.com\/blog\/2025\/02\/01\/coding-your-own-tools\/","title":{"rendered":"Coding your own Tools"},"content":{"rendered":"\n

We’re working on finalizing the 2nd Edition of Algorithms for Optimization<\/a>. We originally set out to add three new chapters, but in addition to that overhauled most of the book. Its pretty exciting!<\/p>\n\n\n\n

These projects are so big, and so long, that you’ll inevitably run into all sorts of new challenges because time has passed, some dependencies are no longer supported<\/a>, you try to do things better \/ a different way<\/span><\/span>e.g., use Docker to compile the book<\/span><\/span><\/span><\/span>, or the MIT Press requirements have changed. That last one is the inspiration for this blog post. <\/p>\n\n\n\n

I ended up writing some new tooling to support alttext<\/a>. Short for alternative text for images<\/em>, alttext is textual content that can be associated with an image and read out to someone using a screen reader. It wasn’t part of the submission materials when we wrote Algorithms for Optimization<\/em> v1 and Algorithms for Decision Making<\/em>, but this time around, MIT Press asked that we supply alttext for every figure in a big spreadsheet. New challenge!<\/p>\n\n\n\n

Mykel<\/a> and I are somewhat different when it comes to being textbook authors. Most authors submit large Word documents with disparate images and let the MIT Press team handle the final text layout. Not us. We provide the final printable PDF<\/a>.<\/p>\n\n\n\n

Our setup is quite nice. It is all under source control, we have a ton of control over how everything looks, and we have everything for the book in one place.<\/p>\n\n\n\n

When I saw the ask to supply this additional spreadsheet, I instantly became worried that having a separate sheet could cause problems. That sheet needs to be kept in-sync with the textbook — if any figures are added or removed, we want to make sure they are also added or removed from the sheet. The sheet is also a somewhat inconvenient place to write the alttext. Ideally it would be defined in the LaTeX documents, alongside the figure that it describes. Most importantly, we need to know if we’re missing any alttext.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Storing Tests by Code<\/h2>\n\n\n\n

We already have some nice technology<\/a> in our textbook-writing workflow that lets us use the algorithms that we present to the reader to both generate figures and author + execute unit tests.<\/p>\n\n\n\n

We present our algorithms using Pythontex<\/a> and algorithm<\/code> environments:<\/p>\n\n\n\n

\\begin{algorithm}
  \\begin{juliaverbatim}
    diff_forward(f, x; h=1e-9) = (f(x+h) - f(x))\/h
    diff_central(f, x; h=1e-9) = (f(x+h\/2) - f(x-h\/2))\/h
    diff_backward(f, x; h=1e-9) = (f(x) - f(x-h))\/h
  \\end{juliaverbatim}
  \\caption{...}
\\end{algorithm}<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
The juliaverbatim<\/code> blocks get typeset, but they aren’t executed.<\/p>\n\n\n\n
We have a script that parses our source files for algorithm<\/code> blocks and exports the juliaverbatim<\/code> contents into a big Julia source file belonging to an Alg4Opt.jl Julia package. We can then load this package when executing Pythontex blocks that do execute, for generating our figures.<\/p>\n\n\n\n
We have had unit testing since the beginning. When we first wrote Algorithms for Optimization<\/em>, we had the unit tests in a separate directory, written in the test files for the Alg4Opt.jl Julia package we exported to. That worked, but the tests were written in an entirely different place than the methods. Sound like storing alttext somewhere other than the figures?<\/p>\n\n\n\n
We ended up defining a no-op LaTeX environment:<\/p>\n\n\n\n
\\excludecomment{juliatest}<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
and then add those after every  algorithm<\/code> block:<\/p>\n\n\n\n
\\begin{juliatest}
let
  for (f,x,\u2202) in [(x->x,    0.0, 1.0),
                  (x->x,    1.0, 1.0),
                  (x->x,    1.0, 1.0),
                  (x->x^2,  0.0, 0.0),
                  (x->x^2,  1.0, 2.0),
                  (x->x^2, -1.0,-2.0)]
    @test isapprox(diff_forward(f, x), \u2202, atol=1e-6)
    @test isapprox(diff_central(f, x), \u2202, atol=1e-6)
    @test isapprox(diff_backward(f, x), \u2202, atol=1e-6)
  end
end
\\end{juliatest}<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
We then parse the LaTeX source files in the same way we do for the algorithm<\/code> blocks, and export the contents of any juliatest<\/code> block as unit tests.<\/p>\n\n\n\n
Storing the tests next to the algorithms makes things a lot nicer. <\/p>\n\n\n\n
Pulling Alttext<\/h2>\n\n\n\n
I decided that I could do something very similar for alttext. <\/p>\n\n\n\n
I defined a dummy command that like juliatest<\/code> does nothing when compiling the book, but lets us put the alttext content into it:<\/p>\n\n\n\n
\\newcommand{\\alttext}[1]{}<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
We can then use it in the source code to define the alttext alongside the figure:<\/p>\n\n\n\n
\\caption{
  A one-dimensional optimization problem.
  Note that the minimum is merely the best in the feasible set---lower points may exist outside the feasible region.
  \\label{fig:one-d-opt-prob}
  \\alttext{A line chart with a single undulating curve and an interval 
           containing a local minimum identified as the feasible set.}
}<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
I then wrote a script that runs through our source files and finds all figure<\/code> and marginfigure<\/code> blocks, and searches for such a command. If it finds it — great, we can pull out the alttext content and export it to that spreadsheet we need. If not, we can print out a warning that that figure (whose \\label<\/code> ID we also extract), is missing alttext. A nice, simple scripted solution.<\/p>\n\n\n\n
Expand this to view the full script.<\/summary>\n
<\/p>\n\n\n\n
using Printf\n\nmutable struct FigureEntry\n    file_index::Int    # Index into chapter files\n    line_index_lo::Int # Index into the chapter's lines at which the \\begin resides\n    line_index_hi::Int # Index into the chapter's lines at which the \\end resides\n    label::String      # Figure label, as defined by \\label command (or empty)\n                       # Figures may not have labels if they are in solutions or examples.\n    alttext::String    # Alt text, as given by an \\alttext command (or empty)\n                       # Every figure is expected to have alttext for the final deliverable.\nend\n\nfunction is_start_of_block(str, block)\n    return startswith(str, \"\\\\begin{$block}\")\nend\n\nfunction is_end_of_block(str, block)\n    return startswith(str, \"\\\\end{$block}\")\nend\n\nfunction get_files(; chapter_regex::Regex = r\"include\\{chapter\")\n    retval = String[]\n    for line in readlines(\"optimization-chapter.tex\")\n        if occursin(chapter_regex, line)\n            m = match(r\"chapter\/\\S*(?=\\})\", line)\n            @assert isa(m, RegexMatch)\n            push!(retval, m.match*\".tex\")\n        end\n    end\n    return retval\nend\n\nfunction find_matching_paren(str::String, starting_index::Int=something(findfirst(isequal('('), str), 0))\n    @assert str[starting_index] == '('\n    nopen = 1\n    i = starting_index\n    n = lastindex(str)\n    while nopen > 0 && i < n\n        i = nextind(str,i)\n        nopen += str[i] == '('\n        nopen -= str[i] == ')'\n    end\n    return nopen == 0 ? i : -1\nend\n\n\"\"\"\nFind the text for a label, such as \"fig:gradient_descent_rosenbrock\" from\n\\\\label{fig:gradient_descent_rosenbrock}\n\nThere should only ever be one \\\\label entry. In the event that there are multiple,\nthis methods returns the first one.\nIf no label is found, this method returns an empty string.\n\"\"\"\nfunction find_label(lines, line_index_lo::Int, line_index_hi::Int)::String\n    for line in lines[line_index_lo:line_index_hi]\n        m = match(r\"\\\\label\\{([a-zA-Z0-9_:\\\\-]+)\\}\", line)\n        if isa(m, RegexMatch)\n            return m[1]\n        end\n    end\n    return \"\"\nend\n\n\"\"\"\nFind the alttext for a figure, which is contained inside an \\\\alttext{} command.\nThere should only ever be one \\\\alttext entry per figure. In the event that there are multiple,\nthis methods returns the first one.\nIf no alttext is found, this method returns an empty string.\n\"\"\"\nfunction find_alttext(lines, line_index_lo::Int, line_index_hi::Int)::String\n    for line in lines[line_index_lo:line_index_hi]\n        m = match(r\"\\\\alttext\\{([^}]+)\\}\", line)\n        if isa(m, RegexMatch)\n            return m[1]\n        end\n    end\n    return \"\"\nend\n\nfunction pull_figures()\n    is_start_of_ignore = str -> is_start_of_block(str, \"ignore\")\n    is_start_of_figure = str -> is_start_of_block(str, \"figure\")\n    is_start_of_marginfigure = str -> is_start_of_block(str, \"marginfigure\")\n    is_start_of_relevant_block = str -> is_start_of_figure(str) || is_start_of_marginfigure(str) || is_start_of_ignore(str)\n\n    figures = FigureEntry[]\n    for (file_index, filepath) in enumerate(get_files())\n        filename = splitext(splitdir(filepath)[2])[1]\n\n        println(\"\\treading \", filename)\n        lines = [replace(line, \"\\n\"=>\"\") for line in open(readlines, filepath, \"r\")]\n\n        counter = 0\n        i = something(findfirst(is_start_of_relevant_block, lines), 0)\n        while i != 0\n            block = is_start_of_ignore(lines[i]) ? \"ignore\" :\n                    is_start_of_figure(lines[i]) ? \"figure\" :\n                                                   \"marginfigure\"\n            j = findnext(str -> is_end_of_block(str, block), lines, i+1)\n\n            if block != \"ignore\"\n                label = find_label(lines, i, j)\n                alttext = find_alttext(lines, i, j)\n                push!(figures, FigureEntry(file_index, i, j, label, alttext))\n            end\n\n            i = something(findnext(is_start_of_relevant_block, lines, j), 0)\n        end\n    end\n    return figures\nend\n\n# Find all figure and marginfigure blocks\nprintln(\"Pulling all figures\")\nfigures = pull_figures()\nfor (i_figure, figure) in enumerate(figures)\n    @printf \"%3d %2d [%04d:%04d] %s\\n\" i_figure figure.file_index figure.line_index_lo figure.line_index_hi figure.label\n    println(\"      $(figure.alttext)\")\nend\n\nn_figures_missing_alttext = sum(fig.alttext == \"\" for fig in figures)\nif n_figures_missing_alttext > 0\n    println(\"MISSING ALT TEXT!\")\n    files = get_files()\n    for (i_figure, figure) in enumerate(figures)\n        label_text = figure.label\n        if label_text == \"\"\n            label_text = \"UNLABELED\"\n        end\n        @printf \"%2d %s in %s\\n\" i_figure figure.label files[figure.file_index]\n    end\nend\n\nprintln(\"\")\nprintln(\"$(length(figures) - n_figures_missing_alttext) \/ $(length(figures)) figures have labels\")\nprintln(\"Good job!\")<\/code><\/pre>\n<\/details>\n\n\n\n
<\/p>\n\n\n\n
The Joy of Coding your own Tools<\/h2>\n\n\n\n
That’s what this blog post is really about. The fact that you can dig in and code your own solution. We spend so much time coding for big company projects, that it is easy to forget that we can code small, useful things for ourselves.<\/p>\n\n\n\n
The coding we did here is not particularly clever, nor particularly difficult, nor particularly large. That isn’t the point. The point is that we had a problem, and we were able to solve it ourselves with software. Our tools of the trade were brought to bear on our own problem.<\/p>\n\n\n\n
I don’t often use coding to solve my own problems, but it does happen every so often. I used coding to create placecards for my wedding<\/a>, for example, and to create the wedding website. I’ve written code to generate .svg files for CNC laser cutters, in order to craft a loved one a nice birthday present. In high school, I wrote a basic notecard program for practicing my French vocab. That one was super useful.<\/p>\n\n\n\n
I am a big fan of Casey Muratori of Handmade Hero<\/a>  (and Computer, Enhance!<\/a>), which gave rise to the handmade movement<\/a>. The ideas there are very similar — there is joy to be had from building things yourself, and you are smart enough to dive into something and learn how it works. <\/p>\n\n\n\n
Anyhow, I think its nice to be reminded of all this from time to time. Happy coding!<\/p>\n","protected":false},"excerpt":{"rendered":"
We’re working on finalizing the 2nd Edition of Algorithms for Optimization. We originally set out to add three new chapters, but in addition to that overhauled most of the book. Its pretty exciting! These projects are so big, and so long, that you’ll inevitably run into all sorts of new challenges because time has passed, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2998","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/posts\/2998","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/comments?post=2998"}],"version-history":[{"count":26,"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/posts\/2998\/revisions"}],"predecessor-version":[{"id":3024,"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/posts\/2998\/revisions\/3024"}],"wp:attachment":[{"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/media?parent=2998"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/categories?post=2998"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/timallanwheeler.com\/blog\/wp-json\/wp\/v2\/tags?post=2998"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}