(Post theme: Storybook Love by Mark Knopfler and Willy DeVille)

More Filmspotting catch-up. Hard to resist a Top 5 list that involves books, obviously!

I’m using roughly the same criteria as Josh & Adam (and Kristen) – in particular: anything that already has a film (even if it’s terrible) is excluded, and it has to be a full-length book. The latter is particularly challenging: in my opinion, short stories & novellas make the best page-to-film adaptations. Full-length books that would translate well to the screen are almost always going to be better as TV series. I’ve also avoided cheating this by suggesting a trilogy or series of films, although some of these could obviously have sequels.

There’s quite a bit of recency bias here. I’ve read a lot of books, but find it difficult to think back over which ones would work particularly well as films. I skimmed over my Goodreads list, but only recent reads have (my) ratings, and it’s tricky relying on average ratings – plus until recently I didn’t add books that I read but didn’t own (library, friends, family, etc).

5. Pastwatch: The Redemption of Christopher Columbus by Orson Scott Card. I’m generally wary of time travel stories (but am a sucker for time loops) although there are some great film examples. Ender’s Game was a terrible adaptation, so there’s a lot of risk here. (I’m aware of Card’s politics and religion, and my disagreement with most of it doesn’t change my enjoyment of his writing). There are some elements to the story that would play very well today: particularly, the environmental aspect, and the reality of Columbus’s impact on the people already living in the Americas. However, the titular redemption might be an issue. Gina Prince-Bythewood to direct (or maybe Roseanne Liang?), introducing new actors for the main cast, with either Justin Baldoni or Chris Evans to star as Columbus.

4. The Lies of Locke Lamora by Scott Lynch. This would need a careful screenwriter, to emphasise the heist/con story and ensure the fantasy element is almost incidental (until it’s critical to the story). It would have to be a long film, but you could probably sacrifice some parts of the story that lead towards the sequels without losing what makes it a great story. Apparently Warner Bros. were working on an adaption, and after that fell through someone else was, but as far as I know there is nothing announced, so it’s up for grabs in this list. Ajay Friese as Locke, Ian Samuels to direct.

3. The 7½ Deaths of Evelyn Hardcastle by Stuart Turton (first published as The 7 Deaths of Evelyn Hardcastle). Apparently Netflix were going to make a series based on this, but gave up, so that frees it up for the list. I mentioned my love of time loops already, and this would be a great addition, mixed in with the Christie-esque aspects. I didn’t love the ending of the book so I feel this could be one of the rare examples of a film that’s better than the source material). Maybe Mark Ruffalo as the protagonist, but I don’t have a director in mind.

2. The Loneliest Girl in the Universe by Lauren James. I love the genre twist in the middle of this story, and it would translate well to the screen, although you’d need to ensure that marketing (especially trailers, which have become awful in recent years) didn’t ruin it. There’s a lot of written communication in the story, and not seeing one of the characters is important, but a good screenwriter could solve that. Lauren James fancast this herself, so it’s hard to argue for anyone else to star. To direct: Gary Ross (bringing elements from Pleasantville, The Hunger Games, and Ocean’s 8).

1. Beak of the Moon by Philip Temple. It’s one of my favourite stories and definitely my favourite NZ novel. It would make an amazing animated film, made entirely in Aotearoa with local talent. There are two huge challenges: firstly, it’s a long book – but I think a good screenwriter (keep Walsh and Jackson as far from this as possible!) could solve that and the imagery would do a lot of the work. More challenging is that all the main characters are male, and this is central to the story (and, in my limited understanding, to kea behaviour). I think you’d want relatively unknown actors to lead (Māori, in their 20s) although I could see cameos by Rhys Darby, Bret McKenzie and Jemaine Clement as the kākā, Taika Watiti as the kākāpō, Sam Neill as the albatross, and maybe Cliff Curtis as Glintamber. I have no idea who could direct – it should be someone local, and someone who can do animation well, which means someone new.

Honourable mentions (in no particular order):

• The Crystal Cave by Mary Stewart. I’m not sure the world really needs another Arthurian film, even if it is based on one of the best adaptations. It’s also probably too long to make a good film and The Hollow Hills would really need to be made as well. I’d still love to see it done well, and I think we’re at a point where there enough people that can do historical fantasy on screen very well. This did have a TV series adaptation, which probably excludes it (I haven’t seen that).
• Daughter of the Empire by Janny Wurts and Raymond E. Feist. Easily the best trilogy in the Riftwar Cycle, and Mara’s story, particularly in this first book, would easily be as compelling as Katniss, Tris, and Teresa were in the 2010s – and we’re due for a resurgence in great YA films (with less dystopia this time). However, I’m not sure you could do justice in a film, and it might have to be a TV series.
• Project Hail Mary by Andy Weir. Better than The Martian, which was a decent film. However, this is apparently going into production in 2024.
• Gideon the Ninth, by Tamsyn Muir. I think the fantasy/sci-fi mash-up, the LGBTQ+ aspects, the humour, and the action would all combine to make a great film. I wonder if it would be too confusing to tell in under 2 hours and that you’d need to make this a TV series instead. It also feels like it would really require also making the other books in the series (once they’re all finished) and some of those would be even more difficult, and that also disqualifies it from this list.
• Faerie Tale by Raymond E. Feist. Problem #1: I remember loving this story and re-reading it several times, but I haven’t read it in over 10 years, so it’s hard to recollect enough details to evaluate whether it would translate to the screen well. Problem #2: I watch hardly any supernatural thriller films, so don’t have any idea what it would be competing with. My gut feeling is that it would be great, though.
• Memory Man, by David Baldacci. This would clearly make a great film – it was only bumped off my top 5 because there are already a lot of great crime films and I’m not sure that this would add anything truly new.
• Hunting Party by Elizabeth Moon. This would be a great action/sci-fi story (much of the best sci-fi on film is space opera). I don’t have any reason for not including this, except that I couldn’t figure out which of my 5 top I would bump to make this. It’s a clear #6.

(Post Theme: Lost the Breakup by Maisie Peters)

Last Friday (30th June, 2023), was my last day working for N-able after 18 years (a bit over 12 years of that for SpamExperts, then after the acquisition a bit under 4 years for SolarWinds MSP, then after the split the last almost two years for N-able).

I’m more able to write about things now, and have a lot of ideas of topics I’d like to cover, and hope to get to a bunch of those before starting my new job in September. For now, mostly for my own record, a brief summary of how it all ended.

Scaled Agile & the introduction of a Product Owner role

In 2020, SolarWinds decided to adopt the Scaled Agile framework (SAFe), under the name “SolarAgile”. The plan was to introduce this within the MSP side of the business first, and then extend it to “Core” (the traditional, large, part of SolarWinds) and “Cloud” (Pingdom, AppOptics, Loggly, Papertrail). I was part of the pilot group helping to shape this in the initial adoption, which was actually pretty great.

One aspect of this project was the introduction of a Product Owner role. There were a few POs across different products, but no real consistency about what they did, and most teams didn’t have a PO. For Mail Assure & SpamExperts, I handled the PO role in the initial pilot (one of the requirements was someone that was both extremely familiar with the product and also very experienced with Agile), but the intention was that I would move to PM after the pilot period.

By this time, the Mail Assure & SpamExperts engineering team had shrunk in size quite a bit (down to two teams), and wasn’t expected to grow again. Similarly, from a product perspective, the goal was high retention rather than a lot of growth (especially new logo growth). I argued that we did not need both a PO and a PM for a product of Mail’s size (a “practicality beats purity” type argument). However, TPTB went ahead and hired someone for the PO role anyway.

In some ways, the acquisition (August 2017) was the beginning of the end – particularly since SolarWinds and then N-able were both very focused on people in offices (covid disrupted this as it did everywhere, but even now there is strong encouragement to be back at an office at least part of the time). However, I feel like adding in the PO role to the Mail team was the real beginning of the end, because it was always clear to me that we were ‘overprovisioned’ in product people for the nature of the product.

Everything went well enough for almost a year – a lot of that time was our new PO learning the ropes while I learnt more about the parts of PM that I hadn’t done before, and tried to extend out our strategic planning.

Reorganisation and Product Owners transformed to Product Managers

In late 2021, after the split from SolarWinds, N-able had another one of its frequent leadership changes, with a new CTO followed not long after by a new VP of engineering and a new VP of product. This (mostly) spelt the end of SolarAgile, and the decision was made to completely remove the PO position, with the actual PO work being brought under the responsibility of the PMs.

Essentially overnight, all the POs were now either PMs or moved elsewhere within the company (e.g. into an engineering role) – many have since left N-able. This transition was handled very poorly, with the news of the change arriving to at least some of the POs at the same time it arrived to the company as a whole.

Most relevant to my journey, this meant that Mail Assure & SpamExperts now had two PMs, the former PO and me. I already felt like both a PO and a PM was too much ‘product’ for the team, and this obviously made it that much worse.

Again, everything went ok for some time – while we were both PMs, we did still tend to split up the work in similar ways as before, with me focusing on the higher level strategy and prioritisation and my fellow PM working on more of the operational aspects.

Interlude: Promotion and Pay

During 2022, I was promoted to Senior Product Manager (although as a contractor I technically didn’t have any title at all). This had a reasonable pay bump, which was particularly welcome in a year where inflation was suddenly something that was relevant again.

This was the third significant pay increase that I received in my SolarWinds & N-able journey. The first was after SpamExperts was acquired – this was basically adjusting to be closer to ‘market’ rates (SpamExperts being fairly small was generally at the lower end, although it had certainly improved and in the last 5 or so years there I had no complaints about how much I was paid), and also essentially for retention purposes (ensuring as many key people as possible stayed after the acquisition). The second was a couple of years later, after the former SpamExperts CEO & CTO had left, and I was very unhappy in my architecture role and with the direction of the product. This was again basically a retention play (which worked – I also got other changes, which helped, but for the most part I was willing to stick around for a while because of the comparatively high renumeration).

It was never made super explicit, although it came up in some conversations, but I feel that these actually ended up contributing to the end (of course, in the meantime, they contributed plenty to my financial security). If you looked at the cost of PMs for the Mail team, then I was the bulk of that, and my salary was somewhere in the ballpark of 2-3% of the product’s revenue. When I moved product (more on that below) this would have been even more noticeable, because the strong retention motivation was no longer there (or at least no longer visible to the people making the decisions)

This isn’t a complaint about being well paid for a few years, by any means. But I do think that it was one factor in the eventual end of my tenure.

Moving Products

Around August 2022, it was clear that people had looked at the budgets for Mail and decided that having two PMs was not justified. I don’t disagree with this (and indeed had argued it all along), although I think things would have played out pretty differently if we’d never had a PO at all (which is certainly not her fault, and is not to denigrate any of the excellent work she did).

Either I would need to move or the other PM would need to move. It was made clear to me that the preference was for me to move – as a senior PM who was well regarded by many in N-able across several products (although certainly not universally; you can’t please everyone), the expectation was that it would be simpler to put me in a new product and have the other Mail PM continue with Mail Assure and SpamExperts.

I didn’t like this plan. I did have, in my performance planning documentation, some statements around either joining a new product or building something new within the same product & team. However, those were in the 3-5 year range, and I was pretty clear about having some work that I really wanted to finish with Mail first.

(As an aside: I never really got the chance to properly implement a strategy as Mail PM. The first 12-24 months were taken up with (a) work on security as a result of the SolarWinds Hack, (b) work to adjust as a result of the SolarWinds/N-able split, and (c) finishing off and cleaning up after the previous PM, who really didn’t understand the product at all. After that, we were just starting to get underway with some new work, and then I was moved away).

However, it was clear to me that me moving was going to be best – it would hopefully turn out well for me (wrong, as I came to find out), and for the PM who would stay (somewhat correct), and for everyone else as well (unclear). So I accepted that this was going to happen, and was moved to Cloud User Hub, which was a product born of another acquisition (spinpanel), and had just a month or two earlier launched. More on how that was sold to me and what the reality was another time.

Another Reorganisation

The first couple of months on Cloud User Hub were rough (mostly because of the state of the product and the disastrous launch), but by the middle of October were improving and things were looking up.

At this point, N-able joined many other tech companies and laid off a large chunk (maybe 15%?) of employees, and did a bunch of restructuring as a result. I was kept on, but my immediate manager was gone, and the product was moved to a separate group as well, under the leadership of someone completely new to N-able.

At this point, looking back it feels like a foregone conclusion that this would be it. The product was in a huge mess, and although I worked on improving that and some progress was made, and although I joined after the mess was made, you couldn’t look at my work in Cloud User Hub and see anything like success. In addition, I was now reporting to someone (my manager’s manager in particular) who had no history with me at all, so there was no existing goodwill or understanding of quality work that I had done in the past.

Final 4 Months

On February 28th, I was told that I was no longer required and would have 4 months of notice, finishing up at the end of June.

The official reason was the challenges around location and time zone. To be fair, this was much harder in Cloud User Hub than it had been with the Mail team. The majority of the engineering team were in India (good overlap with NZ), some in the Netherlands (good overlap with NZ, long history of working with people in that time zone), and some in the UK (reasonable overlap with NZ, several years of working with people in that time zone. However, N-able has almost all of the senior management in the US, and combining the US (or Canada, where there are other teams) time zones with the Europe/Asia ones leaves no good times for working with NZ.

For all of the 18 years I was with SpamExperts, then SolarWinds, then N-able, I was extremely flexible around working times (anyone who worked with me would attest to this). Until Cloud User Hub, this was the occasional meeting during the night, and working around 4-6 hours in my evening (which suited me well for many years anyway). After moving to Cloud User Hub, I would regularly have meetings at 3 a.m., 4 a.m., and so on – at least weekly, generally multiple times a week. I made this work, but it wasn’t good for me (or for N-able, really).

Ironically, this was much worse in the December-February period (excepting the week I vanished in February thanks to Cyclone Gabrielle) than later, when there was less need to be meeting with senior leadership and more work with the actual teams, where time zones aligned well enough. Travel to do in-person work (to Edinburgh, for example) was expensive for someone in NZ, though (none of the engineers in India would be doing that travel either).

More to say about my adventures over the last 18 years, but that’s essentially how it all came to an end!

(Post theme: Code Monkey by Jonathan Coulton)

Ben Hoyt has a great post on designing Pythonic library APIs (itself a written version of a talk he gave). I have some thoughts in response:

Style

I love PEP 20, the Zen of Python (I used to have a set of t-shirts I had made that had one koan on each), and I think it’s actually applicable more widely than just code (Python or otherwise). I certainly agree that following its guidelines is a great start.

Ben suggests following PEP 8 (the style guide), I would go further than that:

• Assuming that there’s some sort of CI pipeline, that should include enforcement of a style (ideally auto-correcting to one). Black is the obvious choice here, and it’s (for the most part) following PEP 8, but the most important thing is to have a consistent style where a tool does all the work.
• Shift work ‘left’ of the CI pipeline, and make it easy for contributors, by having your style format of choice part of pre-commit or some similar workflow, and have an appropriate configuration file for that in the source repository.
• Follow PEP 257 for your docstrings (and obviously have good docstrings). IDEs sometimes shove in a bunch of text around return values/types and enforce conventions – I’m less bothered about those, and I think generally they (and the related recommendations in PEP 257) have been supplanted by type annotations in many cases. When other people are using the library, they’ll see these docstrings, and they’re probably also part of your automated reference documentation.
• While on the topic of docstrings, put an example or two in them anywhere it makes sense, and use doctest to make sure that they stay correct.
• Have a style for imports (and use isort or something similar) to automate/enforce that as well. I personally prefer sorting by (length, normalised case alphabetical) with groupings for, from top to bottom, the standard library, third-party libraries (with a blank line between each, ordered approximately by how well established they are), internal libraries, and then finally imports from within the same package. But again, it’s the consistency that matters most. (This one isn’t really about API design).

“Pythonic”

In addition to the items that Ben mentions, I think it’s important to design the API so that it works well with Python idioms. Exactly what this entails depends a lot on the specifics of the API, but for example:

• Functions & methods should be designed so that they can easily be used with the functools module (Ben has an example of this).
• Provide generators rather than returning a tuple or list when possible. These should work well with the itertools module, with yield from, etc.
• Work well with the standard library logging module (but don’t be noisy when someone isn’t using it). The logging module is an example of an API in the standard library that is poorly designed (or perhaps just is not particularly Pythonic), in my opinion, but it’s the default choice for logging and utilised by tools like Sentry.
• Context managers. For example, if your object connects to something (a file, over a network, etc) then have the cleanup done in a method called close() so that you can using contextlib.closing (but actually also provide your own __exit__ to handle this).
• Where appropriate, make it easy to serialise data. This might include supporting pickling objects, but might also be to other formats (JSON, YAML, etc).

Async

The Python releases that I really like are the ones that focus on improving performance (sometimes this is CPython specific) and usability (like the improved tracebacks in 3.11), and the standard library. In my opinion, for the most part, the Python language itself does not need regular changes, and sometimes these can be at the detriment of some of the aspects of Python that make it great (like readability and ease of learning).

I’m not (yet?) a fan of the walrus operator or pattern matching, for example. I have mixed opinions about type annotations. However, one change to the language over the last decade that I feel is definitely worthwhile is the addition of async & await. It was possible to use coroutines in Python previously, and understanding how async works does add complexity to learning the language, but I feel it expands what can be easily done with the language, in the same way that you can use Python in a very object-orientated way, or a very functional way, and so on.

One catch with async & await is that they have a tendency to spread throughout your code. You can ‘collapse’ async code into a synchronous action by adding an event loop and waiting until everything is done, but for the most part if you’re using an async library then you’re probably building your entire app in an async fashion. It’s definitely much simpler to make use of a synchronous call inside of an async method than vice-versa.

There are libraries that have added async functionality after originally being synchronous (e.g. Django) but from what I’ve heard that has been complicated to get right and the API is less natural than otherwise (e.g. compare with FastAPI).

Whether or not a library should have predominately async methods, or both async and sync versions, or avoid async entirely depends a lot on what it’s doing and how it’s expected to be used. However, it’s definitely something to think a lot about in advance, rather than try to adjust mid-flight.

Exception Chaining

Ben has a good section on errors and exceptions. The only thing I would add is that you can and should explicitly chain exceptions so that it’s clearer to the caller what’s happening. I think Ben is referring to that here, but doesn’t call it out explicitly:

For example, if your library can raise ssl.SSLError when it’s calling an HTTP API, it’s probably best to catch that and re-raise as fishnchips.NetworkError.

Ben Hoyt

Instead of doing this:

try:
    ...
except ssl.SSLError:
    raise fishnchips.NetworkError()

You should do this to show that the SSLError was the “direct cause” of the NetworkError:

try:
    ...
except ssl.SSLError as e:
    raise fishnchips.NetworkError() from e

Related to errors & exceptions, you should make use of the warnings module and the various warning Exception classes.

In this section, Ben also says:

APIs should be designed so that it’s hard to make mistakes.

He doesn’t have this as one of the takeaways, but I would ‘promote’ it to one.

The standard library unittest package

Ben has unittest as an example of a poor standard library API. I agree with this, and I think it provides a good example of where API design can be challenging. The main problems with the unittest API (such as assertEqual(a, b) not being assert_equal(a, b) or a plain assert a == b) come from the API being an implementation of the xUnit API (originally SUnit and popularised by Java’s JUnit but extremely widely used).

The question here is how closely the Python xUnit implementation should match the implementations in other languages (or, put another way, whether the Python standard library should have an xUnit implementation versus a library that supports unit tests that isn’t necessarily xUnit). If you’re coming to Python’s unittest module from Java (and JUnit) then these are comfortingly familiar:

import static org.junit.jupiter.api.Assertions.assertEquals;

import example.util.Calculator;

import org.junit.jupiter.api.Test;

class MyFirstJUnitJupiterTests {

    private final Calculator calculator = new Calculator();

    @Test
    void addition() {
        assertEquals(2, calculator.add(1, 1));
    }

}

import unittest

# Probably a relative import in practice.
import example.util


class MyFirstUnittestTests(unittest.TestCase):
    def setup(self):
        self.calculator = example.util.Calculator()

    def test_addition(self):
        # In Python 2, this could have been assertEquals()
        self.assertEqual(2, self.calculator.add(1, 1))

There are lots of other examples of APIs like this, where some underlying functionality is being exposed in many different languages, or where similar functionality is being implemented in many different languages. Sometimes, you get both, like with the python-mysql library:

import contextlib

import MySQLdb

# Fairly Pythonic, and aligns with the DB API (PEP 249)
# The Connection object should have an __exit__ that closes.
with contextlib.closing(MySQLdb.connect(**[connection args])) as db:
    with db.cursor() as c:
        c.execute("SELECT col1, col2 FROM tbl")
        for col1, col2 in c.fetchall():
            pass

# Low level API that essentially exposes the MySQL C API.
import _mysql as mysql

conn = mysql.connect(**[connection args])
conn.query("SELECT col1, col2 FROM tbl")
result = conn.store_result()
for col1, col2 in result.fetch_row(maxrows=0):
    pass
conn.close()

In general, I believe it’s better to design your API to match the language, and copy the intentions and outcomes from the source, rather than try to completely match the API. This is one reason why pytest is superior to the standard library unittest.

The standard library csv package

Ben has the csv module as an example of a good API, and I generally agree, and I think it’s particularly so given how poorly defined the CSV format is, which makes working with CSV files much more challenging. The one nit I have is:

import csv

with open("some.csv", newline="") as f:
    reader = csv.reader(f)
    for row in reader:
        pass

I don’t love that you have to know/remember to open the file with newline=”” (in Python 2, you had to remember to open the file in binary mode). Most of the time it won’t make any difference, but when you have a CSV with a newline in a quoted field it’ll break if you don’t do this (I’d argue that if you have that sort of CSV you perhaps are using the wrong serialisation format, but often that’s out of your control).

It’s more Pythonic to create objects from files than filenames (more technically: pass something file-like that supports the file protocol rather than something that supports the string protocol). It does feel like passing a string to csv.reader is generally wrong (you’ll get each character in the string as a one column row), and the first argument to csv.reader can already be a list (or similar) or file (or similar), so perhaps a string could be taken to mean a filename. csv.reader_from_filename doesn’t seem Pythonic, or csv.DictReader.from_filename. Having csv.reader call reconfigure() on the passed object is probably a bit too magic (explicit is better than implicit!).

In summary, this is a (very small part) of the csv API that I don’t like, but I don’t have a good suggestion for solving it, either.

from library import something

Ben says:

Takeaway: Design your library to be used as import lib ... lib.Thing() rather than from lib import LibThing ... LibThing().

Ben Hoyt

I agree with the majority of his post but a huge 100% to this specifically. Namespaces are one honking great idea!

Global configuration and state

In this example code of Ben’s, he argues against having a module-level DEFAULT_TIMEOUT:

DEFAULT_TIMEOUT = 10

def order(..., timeout=None):
    if timeout is None:
        timeout = DEFAULT_TIMEOUT
    ...

This is an interesting argument, and goes against the common refrain (probably originating from languages like C) that you shouldn’t have ‘magic’ numbers in your code and should define them at the top level instead.

If your intention is that people should be able to change the default, then you should definitely do this differently (e.g. as Ben describes). If your intention is that this default is never to be changed, then you can make this a little better in modern Python (3.8+):

import typing

_DEFAULT_TIMEOUT: typing.Final[int] = 10

def order(..., timeout=_DEFAULT_TIMEOUT):
    ...

However, you’ll need some sort of type checker to validate that it’s actually not changed (which does go along with Python’s “consenting adult” type approach to things like private variables).

Type Annotations

Ben has a good section on type annotations, which pretty much exactly matches my feelings. I don’t love them, or feel that they are themselves really Pythonic (although their use in libraries like Pydantic does, somehow). I completely agree with Ben’s takeaway:

On balance, I definitely think it’s the right thing to do in 2023 to ship your library with type annotations.

And of course, don’t just use them, but run Pyright or MyPy over your library’s code on every commit.

Ben Hoyt

Ben’s last example of a positive of type annotations is:

They help your IDE provide better navigation and auto-completion.

I agree that this is the case, and one of the more compelling reasons to use type annotations, and also why there’s – to a certain extent – an obligation on library/package developers to provide them. However, I find it generally disappointing. I strongly feel that this functionality should be provided by the IDE without the user needing to put in all the manual work of explicitly typing everything. I wish we had solved this need with better tools rather than by putting a heap of additional work on developers – and especially without adding a heap of boilerplate to Python code. I understand that the dynamic nature of Python makes this hard, but hard problems are good ones to solve.

Overriding Operators

Ben gives a rule of thumb:

Only override math operators like a+b if you’re creating a number type.

Ben Hoyt

I agree with this for almost all math operators, except perhaps + (for example, using + to concatenate strings is more natural than str.join, and I think it was right to make that implementation faster rather than focus all energy on getting people to call join).

I think the standard library has a perfect example of a mistake here, which is pathlib and the division operator. I believe this is a cute hack that is maybe ok in a third-party library, but definitely does not belong in the standard library:

>>> import pathlib
>>> p = pathlib.Path("~")
>>> conf = p / ".mypackage" / "config.ini"
>>> str(conf)
'~/.mypackage/config.ini'

There was a lot of controversy about this back in 2012 (the PEP has a reference to some of it), and maybe the right decision was made, but it’s unPythonic and distasteful in my view.

Keyword arguments

Ben makes an argument for keyword arguments helping with backwards-compatibility, which I agree with. He has this example:

def order(chips=None, fish=None):
    """Place an order.

    Args:
        chips: number of scoops of chips
        fish: number of fish
    """

I would argue that these should be keyword-only arguments. If I’m writing a call to order, I’ll never remember whether fish or chips comes first (I would even argue that they are backwards here, because – at least where I am – people say “fish and chips”, not “chips and fish” – an irreversible binomial if you’re a linguist). An IDE might help out when writing, but when reading the code, you’re not necessarily going to have that context made available. A two character change, but it prevents easy mistakes:

def order(*, chips=None, fish=None):
    """Place an order.

    Args:
        chips: number of scoops of chips
        fish: number of fish
    """

I also worry a bit about how this would scale. A fish’n’chip shop probably has at least a dozen items on their menu, and that’s a lot of arguments. It could be generalised, something like:

def order(**kwargs):
    """Place an order.

    The arguments should be the name of the item, with the value: either the number of the item,
           a tuple of (quantity, type),
           or a list of such tuples

    For example:

    >>> fishnchips.order(chips=1, fritters=(4, 'potato'), fish=[(1, 'crumbed'), (1, 'battered')])
    """

The main concerns I have about this are:

• If you want to have arguments that are not items in the order (like the timeout one Ben has as an example), it feels messy for that to be mixed in with the items. Using **kwargs helps, because you’d make timeout a keyword-only explicit argument and that would distinguish it, but it still feels untidy to mix order items and order configuration into what is essentially one dictionary.
• The item names are limited to what’s supported by Python names. That means it can’t start with a number, can’t have spaces, can’t have punctuation, and so on. For the most part this is probably fine – you can have onion_rings instead of ‘onion rings’ and the like. It feels like it might get challenging to remember the rules for converting from ‘real name’ to ‘argument name’, though. I also suspect that eventually the method will need the ‘real name’ for logging or display or similar.
• Related to the issue of remembering the rules is remembering what can be ordered. With Ben’s version, the signature tells you that you can have fish or chips. With my kwargs one, it does not – presumably there is some validation of the arguments in the function, but that doesn’t help the caller prior to runtime.
• I’m not sure how well this would continue to scale. Say that after adding crumbed fish, we decide to sell both Snapper and Tarakihi. Do we now support an argument like fish=(3, ‘crumbed’, ‘snapper’)? How do I remember that it’s not fish=(3, ‘snapper’, ‘crumbed’)? How are we going to handle burgers, which have lots of ingredients that people might want to customise?

I realise Ben was using this as an example of how keyword args and dynamic typing help backwards compatibility (which they do), and not trying to design the ultimate fish&chips library API. However, I feel like you’d need to move to a single collection of objects (probably dataclasses) before long.

(Post-theme: With a Little Help From My Friends by the Beatles)

LLM & generative AI buzz is everywhere, and assistance writing code (either via something like copilot or by just querying a model) is supposedly a huge productivity boost, so I was curious enough to try this out. Simon Willison has written a lot of interesting things about LLM including how he’s used it for developing, and made it seem like something that could be useful.

As a first experiment, I wanted to use the “ask the LLM to draft something for me” approach (rather than a IDE-integrated tool like copilot), and I wanted to avoid having to go through a ‘conversation’ of prompts (particularly one where I had to guide the model to a solution I already knew or guessed). It’s usually pretty easy to find what I need by looking at the docs (or code), so if this is going to be useful for this sort of work, then I need it to be simple and fast.

Details of the two attempts I’ve made so far are below, but in summary: I’m underwhelmed so far. Maybe I’m not trying the right sort of problems (one was real, one was an exercise), or maybe I need to be better at prompt crafting or more willing to spend time ‘talking’ with the model. I have some investigation/experimentation to do with production use of LLM so may try out some more personal use then.

Redirecting users to a custom landing page in Django

I have a Django site that’s about 13 years old, so has gone through a lot of Django versions. It has the ability to specify a landing page (ie. where you end up after logging in) per-user, if one isn’t specified in the params (ie. in the typical “next” arg). I noticed that at some point this had broken, and wanted to fix/re-implement it for modern Django.

I asked

How do I get Django to redirect a user after logging in to a page specified in a user model field?

The answer I was given is:

You can get Django to redirect a user after logging in to a page specified in a user model field by defining a custom redirect function in your `views.py` file and using it in your `urls.py` file.

In your `views.py` file, define a function that retrieves the desired redirect URL from the user model field and redirects the user to that URL after login:

```python
from django.contrib.auth.decorators import login_required
from django.shortcuts import redirect

@login_required
def custom_redirect(request):
    redirect_url = request.user.redirect_url # Replace 'redirect_url' with the name of your user model field
    return redirect(redirect_url)
```

In your `urls.py` file, map the custom redirect function to the Django login view:

```python
from django.contrib.auth.views import LoginView
from .views import custom_redirect

urlpatterns = [
    # Other URL patterns...
    path('login/', LoginView.as_view(redirect_authenticated_user=True, template_name='login.html'), name='login'),
    path('redirect/', custom_redirect, name='custom_redirect'),
]
```

Note that in the above code, we're using Django's built-in `LoginView` and setting the `redirect_authenticated_user` parameter to `True` to automatically redirect authenticated users to the `custom_redirect` URL. We're also specifying the path to the login template using the `template_name` parameter.

Finally, update your user model to include the `redirect_url` field:

```python
from django.db import models
from django.contrib.auth.models import AbstractUser

class CustomUser(AbstractUser):
    redirect_url = models.CharField(max_length=255, blank=True, null=True)
```

Now when a user logs in, they will be redirected to the URL specified in the `redirect_url` field of their user model.

The immediately obvious problem here is that this doesn’t work. Nothing gets the user to the /redirect/ path, so although that would work to redirect to the user’s chosen page, it isn’t any part of the login flow. The second obvious problem is that the redirect_authenticated_user doesn’t do what the model described at all – it’s basically about bypassing the login page when someone is already logged in.

(Side note: while looking up links for this post, I came across this SO question about this (which is likely forming a lot of the LLM’s answer I suspect). It’s not that clear either, but I guess going to SO is what people typically did before LLM? Personally, my go-to is always the docs.)

I could make a solution like this work – if you set the LOGIN_REDIRECT_URL to be the redirect path and a few other adjustments. However, it’s messy and handling a ‘next’ style param won’t work properly without duplicating a bunch of code.

For reference, what I did (which I believe is the ‘correct’ approach, at least as-of the version of Django I’m using) is:

# Add this line to my URLs.
path("accounts/login/", [...].login.LoginUserRedirectView.as_view()),


# Add this class to a login module in my app.
class LoginUserRedirectView(LoginView):
    """Like the parent, but with a custom per-user redirect system."""
    def get_redirect_url(self):
        # It would probably be cleaner to just check for the Anonymous User.
        if not self.request.user or not hasattr(self.request.user, "last_login"):
            return super().get_redirect_url()
        # Having "next" in the request parameters is already handled by the normal login view and will bypass this view completely.
        redirect_to = self.request.POST.get(
            self.redirect_field_name, self.request.GET.get(self.redirect_field_name)
) or self.request.user.landing_page or ""
        url_is_safe = url_has_allowed_host_and_scheme(
url=redirect_to,
allowed_hosts=self.get_success_url_allowed_hosts(),
require_https=self.request.is_secure(),
)
        return redirect_to if url_is_safe else ""

The documentation on this (unusually for Django) isn’t great. However, the code is very clear, and makes it pretty obvious how to do the appropriate subclassing (as well as some safety checks).

Evaluating a basic maths expression in Python

The second one I tried was from an interview exercise. When I was talking to interviewers after completing the test, one of them mentioned that there was a lot of submissions that were clearly from LLM, which made me curious how it would have done. Using one during the exercise didn’t occur to me, but I doubt it would have appealed anyway, since it would consume a bunch of quite limited time and I wouldn’t have expected it to do particularly well.

The task was to take a string like “3 + 7 / 5 * 2” and give 5.8. No brackets, only integers, only addition, subtraction, multiplication, and division, respecting the precedence of operators.

I asked

Write a Python script that can add, subtract, divide, and multiply given an input string, and respects the order of operators, so multiplication and division are done before addition and subtraction.

The answer I was given is:

Here's an example Python script that can perform basic arithmetic operations while respecting the order of operations:

```
def calculate(expression):
    # Split the expression into a list of tokens
    tokens = expression.split()

    # Initialize the result to the first number in the expression
    result = float(tokens[0])

    # Loop through the remaining tokens and perform the appropriate operation
    for i in range(1, len(tokens), 2):
        operator = tokens[i]
        operand = float(tokens[i+1])

        if operator == '+':
            result += operand
        elif operator == '-':
            result -= operand
        elif operator == '*':
            result *= operand
        elif operator == '/':
            result /= operand

    return result
```

To use this script, simply call the `calculate()` function with a string containing the arithmetic expression you want to evaluate. For example:

```
expression = '5 + 3 * 2 - 4 / 2'
result = calculate(expression)
print(result)  # Output: 10.0
```

This is also very clearly wrong. For one thing, the example at the end has the wrong answer – if you ignore precedence then it would give 6 and if you do it correctly it would give 9 (I’m not sure how to get 10). Looking at the code, it also clearly does nothing at all to respect operator precedence.

More importantly, there’s no code that even vaguely hints at dealing with operator precedence. Also, converting to a float is risky with maths unless you really know the context and understand the implications.

For the record, I considered three approaches to this.

Eval()

The first was trivially simple – use eval(). I spent quite a bit of my allowed time internally debating whether the point of the exercise was to check whether I knew eval() existed and how to appropriately use it and that I wouldn’t pointlessly re-implement built-ins – or whether it was not meant to be used, even though that wasn’t in the (otherwise quite detailed) instructions. I put in a bunch of code to handle errors and additional safety restrictions, but at heart, this is just

return eval(input_string, {}, {})

Double-pass Item Replacement

The second was to do two passes of the input string (first for multiplication and division, and second for addition and subtraction), replacing chunks of the expression by the results. This is fairly straightforward, and my main concerns at the time were that this seems like it would get very messy as soon as you try to extend it to do anything else, and that tracking your location in the expression gets messy when you’re changing the length as you iterate through it.

Roughly, again ignoring error handling and so forth, and ignoring that this has a bunch of unneeded conversion to decimals, this is:

tokens = s.split()
i = 1
while i < len(tokens):
    if tokens[i] == "*" or tokens[i] == "/":
        left = decimal.Decimal(tokens[i - 1])
        op = tokens[i]
        right = decimal.Decimal(tokens[i + 1])
        if op == "*":
            tokens[i - 1:i + 2] = [left * right]
        else:  # op == "/"
            tokens[i - 1:i + 2] = [left / right]
    else:
        i += 2
i = 1
while len(tokens) > 1:
    if tokens[i] == "+" or tokens[i] == "-":
        left = decimal.Decimal(tokens[i - 1])
        op = tokens[i]
        right = decimal.Decimal(tokens[i + 1])
        if op == "+":
            tokens[i - 1:i + 2] = [left + right]
        else:  # op == "-"
            tokens[i - 1:i + 2] = [left - right]
    else:
        i += 2
return tokens[0]

In terms of an exercise, this shows an understanding of replacing a slice of a list in Python, and the performance isn’t terrible (memory is fine, looping twice is ok). There’s a bunch of tidying up that could be done, but it would probably have sufficed. I don’t like that it would get complicated quite quickly if you expanded it – adding parentheses and brackets, for example. It is better than eval, unless the point is to show that you shouldn’t reimplement something unnecessarily.

Convert to Post-fix Notation, Evaluate

The third approach, which is the one I like most, was to convert the expression to RPN and then evaluate the RPN expression. I could remember that RPN evaluation was trivial (from way, way, back in my student days) and also that it was fairly simple to convert from in-fix to post-fix (also something I vaguely remember doing, probably in a Data Structures & Algorithms course ~25 years ago, probably in C++). I remembered that Dijkstra had an algorithm for this and the name had something to do with railways, but not the exact details (I looked it up for this: the Shunting Yard algorithm), and that it was basically just using two lists (I would have had to look up the exact details or play around with this, but for the simple case of just addition, multiplication, subtraction, and division, I could remember enough).

Roughly (without extensive testing or any of the error checking, etc), this is:

# Convert from in-fix to post-fix.
precedence = {"+": 2, "-": 2, "*": 3, "/": 3}
tokens = s.split()
op_stack = []
output_queue = []
for token in tokens:
    if token in "+-*/":
        while op_stack and precedence[op_stack[-1]] >= precedence[token]:
            output_queue.append(op_stack.pop())
        op_stack.append(token)
    else:  # Assume a number for now, ie. we're ignoring parentheses and other things.
        output_queue.append(decimal.Decimal(token))
while op_stack:
    output_queue.append(op_stack.pop())
# Evaluate the post-fix expression.
funcs = {"+": "__add__", "-": "__sub__", "*": "__mul__", "/": "__truediv__"}
i = 0  # A for loop would work here, but I hate changing a list while iterating through it.
while len(output_queue) > 1:
    if output_queue[i] not in funcs:
        i += 1  # Skip this, we'll grab it when we get an operator.
        continue
    i -= 2  # Go backwards to get the two operands.
    left = output_queue.pop(i)
    right = output_queue.pop(i)
    op = output_queue.pop(i)
    output_queue.insert(i, getattr(left, funcs[op])(right))  # This probably tries to be too clever and a simple 4-clause if statement would be fine.
return output_queue[0]

In terms of the exercise, this beats eval (again assuming the point isn’t avoiding pointless work) and I feel it beats the previous version, since it’s more readily extendable. Using the magic methods rather than just an if statement is probably unnecessary, but shows an understanding that they exist, that you should call them against the object, and shows an understanding of using functions as first-class objects (having them in the dictionary). It’s more expensive in terms of memory – it could be improved a bit, but generally it’s creating new containers not adjusting the existing one, so will always be worse. The code could do with a bunch of cleanup, especially the RPN evaluation, but it suffices for something done in a quick exercise.

Performance-wise, with a few trivial test statements (not trying to do this seriously at all):

$ python -m timeit -s "import eval_math as m" "m.test_eval()"
10000 loops, best of 5: 21.8 usec per loop
$ python -m timeit -s "import eval_math as m" "m.test_double_loop()"
20000 loops, best of 5: 10.6 usec per loop
$ python -m timeit -s "import eval_math as m" "m.test_rpn()"
10000 loops, best of 5: 24.5 usec per loop

There are, of course, many other ways to do this.

Conclusion

I’m sure that LLMs can assist with coding, and make me more efficient. I don’t feel I have figured out the way to make that happen, yet. More experimenting to do in the future, I suppose.

(More Filmspotting back episode catching up, although not so far back in time now. Post theme music: Escape (The Pina Colada Song) by Rupert Holmes).

Josh and Adam did this as a draft, and I can’t really do that since I don’t know how my choices would have impacted their choices – although I would have got first pick, guessing Killmonger had 1000 kills (low but much closer than they guessed). So, just a regular top 5.

Similar criteria as they used: this isn’t the most powerful villain – it’s the ones that are the most interesting, and that make their film(s) more enjoyable and compelling.

5. Quentin Beck (from Spider-Man: Far From Home) – it’s done regularly in other films, but generally it feels like the MCU doesn’t do films where the villain starts out (appearing) as a hero and then that turns around (although they did it very poorly with Wanda). Good backstory and no real super-powers (although crazy good tech).

4. Ultron – not my favourite Avengers film, but Ultron as a character is really interesting. More humour than a lot of the villains, reasonable backstory.

3. Loki – a tricky choice, since he’s not really a villain any more, but certainly was originally. So much fun, compelling motivation, great dynamics with everyone else. It’s obvious why he kept coming back.

2. Killmonger – for all the reasons that Adam and Josh outlined. Pretty compelling backstory, really flawed.

1. Helmut Zemo (from Captain America Civil War) – compelling motivation, and achieves so much without having any kind of super powers or vast wealth.

Honourable mentions: like Filmspotting, I only included films, but if the TV series were included, then Kilgrave (from Jessica Jones) would definitely be in my top 2 (Salinger and Trish are also great). Fisk (from Daredevil) would probably make my list as well. Ward (from Agents of S.H.I.E.L.D.) would be in the running as well, especially from the earlier seasons. Arthur Harrow (from Moon Knight) would be one I’d have to consider too.

(Still catching up on back episodes of Filmspotting – from last year, not when they did it in 2018. Post theme: Lady in Red by Chris de Burgh).

5. Chris Pratt – I do like the Guardians movies (less so the second one), but I’m not a big fan of the Jurassic World trilogy. The LEGO Movie and Passengers were good, but nothing else stands out to me.

4. Chris O’Dowd – using the same “5th Chris” as Josh & Adam did. Good in the MCU, but the other things I’ve loved him in are TV, which seems a cheat here (The Big Door Prize, The I.T. Crowd). I did like Juliet, Naked.

3. Chris Hemsworth – really good as Thor (except the latest one). Mixed on Extraction, Men in Black, Huntsman. Disliked Ghostbusters. Wrong era in Home and Away for me to know him from that. A few blind spots here.

2. Chris Pine – the Star Treks were so-so. I liked the first Wonder Woman, but did not like the second. I’ve heard good things about the new D&D movie, but haven’t managed to see it yet. Spider-Verse is one of my favourite movies, but he’s not a big part of that. A lot of other films that are blind spots for me.

1. Chris Evans – easily the best MCU Chris (not just the character, but in the biggest group of the better films). Ghosted was fun, The Grey Man was ok, Lightyear was an odd movie, but I like him in it, hated Don’t Look Up, enjoyed the small cameo in Free Guy a lot, liked Knives Out, liked Gifted, Fantastic 4 was not great.