Rob Ede – Actix Web, Life as an Open Source Maintainer, Supply-chain Security

Choosing dependencies

Tim: And that’s kind of life, going back to this, how do we choose dependencies? I find it really difficult to recommend any specific crate because I feel like in two years, or like, yeah, maybe two years time, I am going to be changing my recommendations. Uh,  it’s weird that the language, the, the idea of, you know, stability without stagnation in Rust kind of doesn’t work.  Doesn’t work  because, the language is backwards compatible, but the dependence on third party crates means that there’s no real stability for programs.  I don’t know if that makes any sense, let’s try again.

Tim: Error handling in Rust is a really fascinating topic and one of the things that strikes me is that it’s very difficult to choose a specific dependency or make a concrete recommendation. And it seems like there’s a lot of churn in the crate ecosystem. Like, is rust going to settle down? Like is the, is is  the,  yeah. Is it gonna be easy to make, is concrete choices about which crate to use for specific tasks?

Rob: Yes, uh, certainly from a difference from back in the failure crate days is that we aren’t particularly bound to, um, that type anymore, right? That was an explicit kind of, um, implementation of a trait for a type in a crate, which was like kind of bad and people didn’t particularly like that. The current choices are a lot better because they’re effectively just helpers to  let you implement the standard error trait in a nicer way, or interact with it in a nicer way.  Um, and so the story from that point of view has like, gotten a lot better. Um, what we’re missing is the parts that these crates add, which is like context. Like, um, and understanding of, you know, particular types of errors without just straight up boxing it into a dyn error, um, which loses you a lot of this valuable context, which you might need down the line in say a web framework, right?

Tim: Yeah, there was a really interesting, sort of slightly tangential, but sort of on the theme of recovering from errors, Nico Matsakis, I think last week or the week before,  maybe, just posted a thing saying, oh, we should actually maybe consider deprecating unwinding, um, on panic. And one of the arguments there is that web frameworks don’t actually seem to use  unwinding to recover from errors.

Rob: Yeah, I have that open in a tab and I haven’t read it yet. Uh,  I mean, that is his main point, yeah, possibly. Uh, it’s true that,  you know, web frameworks, which utilize tokio, uh, don’t do this largely because, um, we are trying to remain slightly fault tolerant and so yeah having a panic somewhere and a handler doesn’t crash your entire application for other people connected to it. Which is kind of a special case here. It’s arguable whether there should be opt in behavior for Tokyo spawns.

Tim: Yeah, the  next question, or at least one that I’ve been wanting to ask, is that you’re writing a back end framework, but I thought that you were sort of a front end person, and  is there like an impedance mismatch between those two domains?

Can you elaborate on an impedance mismatch there?

Tim: I think a better question is, are front end frameworks and back end frameworks so different that people need like a different methodology on both of them?

Rob: No, it’s not something I’ve thought about in great detail in the past. Um,  I think maybe with, uh, there were similarities, certainly when you talk about,  front end frameworks that have like routing built in, you know, it’s one of the key components of a web framework is a router, and most of the time like going from page to page is handled by a router on the front end in the same kind of way with pattern matching on on paths and stuff like that. But perhaps the the needs of the two systems are kind of  fundamentally different in other ways, like security, for example. Um, the, the front end needs a much more strict kind of security boundary in order to remain, um, secure against all sorts of kind of attacks, right, which is what browsers, uh, defend you against with the help of various security headers from the back end. Um, that’s one example I can think of.

Tim: I was wondering, maybe this is a suspicious question to ask, but let’s go there,  like could Actix web grow into like a quasi front end framework, given that you have a lot of domain knowledge, could you sort of bolt in Leptos into it so that, or pick your other winner whereby essentially you would have something that’s much more, lets say full stack, in Actix web by default, let’s say.

Rob: Um, I don’t think that’s worth anyone’s time at this point.  Uh, it seems pretty clear that frontend has, has winners. Um, but it also has 10,000, 10 more each week, right? Um,  and the winners still keep winning. So there’s like an established pattern of, especially when you grow, like I understand, that, yeah, maybe,  previous attempts, I haven’t played with leptos, but like previous attempts at these huge full stack, um, frameworks that try to do both sides and wire things together, right? Um, they don’t work for mid sized corporations and up. They just don’t, um, because there’s some fundamental asynchrony between the releases of back end parts, releases of front end parts, releases of mobile parts, which you can’t control, by the way. You have no control. You have like a one hour control over when your mobile app gets put on the App Store, on Google and Apple. Um, and so if you’re trying to have a simultaneous release of everything that has like a breaking change, oh, but it’s fine. We release everything together, you actually don’t. So,  so you need to think about this, um, a bit more holistically, and to do that, you need independent pieces. And that’s one of my guiding principles and why I don’t think it’s worth anyone’s time right now, anyone’s time regarding Actix web.

Tim: Yeah, so Rust, the Rust way, or at least what has been,  uh, when people write web frameworks, they’ve typically written micro frameworks. Uh, and they ask users, or users in this case are developers writing for end users, the users of the framework say, Oh, bring your own templating library, or bring your own everything else.
And, but that puts a lot of pressure on users to know a lot about the domain and all of the crates before they actually build a thing. Is that  a, uh, is that a concern for you as a web framework developer?

Rob: Um, it’s a concern because it’s the concern of the community at large, I think. And, um, I’m quite happy to explain to people why we don’t include templating or don’t include database drivers. And, um, there’s kind of two parts to this.

Tim: Oh really? I’m quite curious, tell me about templating and database drivers.

Rob: Yeah, I mean, I think everyone keeps saying “Oh- “, no, let me re-phrase that.  There’s a push for a rails like framework as the end game, right?  The end game of maturity of Rust web is we have something that’s like Rails or like Django, and that’s important actually because it helps migration for those people for the for that community

Tim: Yeah, I have a similar goal, in fact I have a prototype, I don’t know if you’ve seen my stub of one.

Rob: I don’t think I have seen this.

Tim: No, it’s um, it’s in, it’s called InterCity, like, because the trains network in Europe is, uh, and  but  I shouldn’t talk about my stuff. I really should focus on. So, yeah, I agree with this goal. I think that an, an,  an end point, although it feels very weird to be talking about endpoints for Rust. when Rails was released, what, 2004? 2005?

Rob: yeah, it’s old.

Tim: Uh, like, are we….? Why are we a decade, l ike, why are we so late to getting, like, a full stack thing? It’s kind of another side question, but yeah, you know, I’m very curious about templating  and database drivers because I just hate making choices, uh, when I, you know, Or at least, I don’t know.

Rob: Um, I don’t like picking winners generally with this framework, I think, well, you have to be careful when you’re picking winners, I think is a more appropriate statement to make about that, right? Cause, Actix web has picked Tokyo as the winner of the async runtime. Okay. Um, it’s picked a couple of other crates as the winner of that kind of, uh, area of expertise.

Tim: Is that, for example, serde? Is, are you with JSON

yeah, sure.

Rob: yeah sure, good point. Um, Serde, of course, is well loved in the community as a whole. No one’s really questioned that part. I’ve never had someone say, why Serde and not, um,  eserde or whatever.  But you do get people occasionally saying, will you expand this to make it frame, er, runtime agnostic? And the answer is no, that’s definitely a winner picked. Um, but like when it comes to templating, the, the reason that I don’t want to pick a winner that is that it’s so easy to integrate your own, that why would I limit the flexibility when I could just give an example of all of them? You can go into the examples repo, see which one looks best in the Rust context, and then copy the, copy the folder if you, if you want to with all the dependencies set up and up to date. Um, that to me seems just as easy as providing a framework with a templating library built in.

Tim: Yeah, sure. And I mean, so, do you think that someone could use Actix Web to, as a way to build something that’s Rails like?

Rob: Yeah, for sure.  Um, I’m familiar with a project that’s come up recently called Loco.

Tim: Yeah, I haven’t counted it, but I haven’t really read much, um, more than its readme.

Rob: No, I haven’t particularly played this at all, but it’s come up and people got sort of very excited about it. And, um, it seems like it’s made a lot of these choices. Um,  yeah, emails, background tasks, it says everything included, so it must be some database somewhere. Um,  And yeah, I’m, I’m a little bit suspicious about this in, in general because I think  maybe in some domain like databases, there isn’t a good winner yet. That’s the other, uh, side of the coin here. We’re still figuring out the interfaces for a lot of these things and most of them aren’t perfect. Um,

Tim: I think that that’s, that’s a very fair point. Like we don’t have something like the jdbc. I think that’s the, or is it the,  The, yeah, is that right? Yeah, for Java and, uh, Python has its own, um, you know, the standard library provides a custom, sorry, provides a, an interface that every, uh, everyone uses and it’s, it’s, it’s not, yeah, yeah, the, uh, there was a, a pip, like a, sort of like an RFC, um, which described an API that you should implement if you’re writing a database driver. And they did.  So  your, your import statement changed and like Rust, sorry, Python doesn’t really have interfaces like in the language, at least I don’t think so. Although I haven’t really been paying a huge attention to the recent changes in Python.

Rob: Yeah, I don’t know.

Tim: But, uh, the,  the problem that sort of Rust faces is that that kind of thing doesn’t really exist, at least. It doesn’t strike me.  And there’s a, there’s a weird problem where there’s very few database drivers. Like this, if you want to do Postgres, sure, SQLite, okay, but surely we- and I mean, actually that’s not quite fair, there’s uh, plenty, like there’s a, there’s a bunch, but I just, it’s surprising how poor the data, like, database interaction system kind of is.

Rob: Hmm…yeah, writing database drivers is hard, right? ,

Tim: But shouldn’t, shouldn’t Rust make that actually  okay? Or at least less awful? Like, be easier than-  I don’t know, I’m asking a lot of Rust.

Rob: I think we’ll start seeing really, really good database drivers when the database  the databases themselves provide one, I guess. Uh, it’s sort of one of these first party problems that you see with other kind of databases like Kafka, um, I think about in particular in that they provide a server and a client and it’s always in sync and you’ve got some of these problems with compatibility if you don’t do that, um, with regards to new features and stuff, and the same is probably true of Postgres, the amount of, um, versions that they release a year is, well, three or more, so,  you’re then tasking someone in every language community to keep up with that.

Tim: It’s tricky.  I guess I suppose there Postgres’ answer would say libqpq, we’d probably keep that up to date, and then hopefully it would kind of bump things downstream. But I suppose the next kind of follow on about this is, do you have any advice for people when they are building an application about which dependencies to check?
Take on, or at least to say like, how would you evaluate things? I don’t, it’s a very broad question. I’m sorry for kind of throwing it at you

Rob: How do you evaluate things? Um,  as an application developer?  It’s largely just down to what you need, um, at the moment, because I think most things exist on crates.io now, um, and sometimes the most downloaded version is the one you want, and sometimes it has been abandoned and is still being used transitively by 10, 000 other things, and you should probably use something else, but that’s really hard to tell, and we, we need better tooling around that kind of choice making. Uh, I think there’s initiatives to do this at the moment, uh, that you hear about on the podcast, so just try and keep your finger on the pulse of this stuff, really.

Tim: Yeah, what you need to do to follow on Rust dependencies is to just basically spend 10 hours a week, uh, consuming content. No, I don’t mean to be too facetious. Are you- actually, it’s kind of, how do you keep up with everything in Rust? Like, I don’t.

Rob: You have to pick your battles really. I keep up with the things that interest me in my open source life and in my work life and I try and read changelogs where possible for relevant releases.

Tim: So if someone released like a binary blob of, I think, I was trying to make a weird like segue to a recent, um…

Rob: If you want to talk about that, I’m quite happy to.

Tim: Okay, sure, let’s go there.

Rob: Let’s bring the..

Tim: Let’s bring it.

Rob: Let’s bring the spicy takes.

Tim: Oh, look, I don’t think it’s such a spicy take as, I think that it is important to be able to have a, like, a more sophisticated discussion than just, oh, I don’t even know how to express it- than spicy takes, there needs to be some, forum to be able to have  a, uh, a way to provoke thinking, that is a little bit beyond just kind of random anger tweet, or like rage tweeting or something.

Rob: Absolutely, yeah.

Tim:  Uh, so do you want to provide the context of like my weird, uh, intro?

Rob: So, I mean, yeah, if a very popular library in among a great set of other very popular libraries on crates.io that you’ve definitely heard of suddenly has, um,  well, actually I shouldn’t even say suddenly. It has a pull request filed against the wrapper that includes some description of a way to make it faster. And this way, um, includes a pre compiled binary, and It passes whatever review system is in place for this repo, it gets released, two months go by, and then people notice. And my question to the audience is, does it really matter then? If you really cared, If you really cared, it would have been spotted much sooner than two months ago. Um, you know, that’s my spicy take, and I think my colleagues disagree, because they’re, uh, they were up in arms, much like a lot of the community was.

Tim: I, you know, I,  yeah. A lot of it was to do with like, this whoa shock. How could you do this? How could you abuse the trust of the community. Um,  I want to say something annoying, which is like, I can sort of see both sides of this. One, I can actually see why pre compilation or like binary distribution  of, uh, some parts of the thing is something that people want. Because if you ask Rust users, what do they complain about? They say your compile times.

Rob: Yeah.

Tim: It’s like, well, you’re a critical dependency inside the dependency path of most Rust projects. If I could speed that up, that would be kind of a good thing. Again, you’re literally, uh, making the world better for most Rust developers at that point. With the kind of the very, very significant proviso, or the asterisk, that, who knows what the hell’s in that binary?

Tim: It’s not reproducible. Nobody else could, nobody has been able to reproduce that thing.

Rob: They’ve been able to produce it within, within like, seven bytes, and the, those bytes were a date.

Tim: Ah, yeah, yeah, yeah, that’s, that’s what I heard. No, no, sorry. I, um, I’m trying to get my conspiracy theorist, uh, happening.

Rob: Yeah, yeah. It was a plant.

Tim: That’s right, that’s right. It’s like, but we didn’t reproduce the reproduce, the reproduction, like show me, um, show me the build system.

Rob: These would be, these would be an awesome set of tools to have. And if we have them, we should absolutely provide them to the community and have, uh, or, or encourage the top developers of, of, or top maintainers to use them. Um, because they are, they’re important and it’s not even, is this maintainer malicious even it’s, um, have they hacked?

Tim: No I think there was genuinely good intent there, right? But yeah, yeah, well someone has the root keys. And  OpenSSL was the same, right? The, uh, it was actually, or it was this different, so the OpenSSL breach that was relatively recent now was  a, oh, it was a parsing library, wasn’t it? Where someone uploaded some testing data like, as like, oh, it was a compression library that was, um,

Oh, yeah, it was, um, yeah, compression.

Tim” And so this was the example data, but in the, in automake and like some weird  obfuscated Perl script could make use of that test data to kind of weirdly inject a backdoor into OpenSSL. So it’s kind of ingenious, but yeah, there,  um,

Rob: I think that was more social engineering and like bad maintainership, of course. But, you know, someone’s got this key, they can get hacked. No one’s security is perfect. So we can’t just assume that these things are malicious all the time.

Tim: I had a tantrum. I had a tantrum that Alice, um, from the Tokio team, like, released, uh, version 1.3.7 live on stage.

Tim: I was like, I  I have, like, concerns about that. Like, I would like very significant crates, et’s say if you were, um, yeah, let’s say that you were this era or something. Well, if you’ve got more than 10 million downloads or something, or some threshold, that we apply a gate at crates.io. that says for some embargo period someone has uploaded a new version you could opt into it if you want but it hasn’t been externally verified and there should be kind of some sort of process for being able to verify that the dependency that you are your users are downloading actually means, like actually is the download that was intended, because we have a large number of dependencies whereby the git tags version of the hashes, don’t match the downloads in crate.io. So, yeah. Anyway, now I’m going on a bit of a rant.

Rob: Of course, no, these verification steps are very important, uh, in the long run. I think as, as more and more people get involved, the, uh, potential threat, um, boundary gets bigger. Right. Um, so these are cool. It’s cool to think about a future where we have verifiability in the stuff we’re uploading. And also, um,  was it the chain of, of sources back to the source code, right?
Um, I’m guilty of this because I happen to use, uh, dash dash allow dirty on publishing everything which somebody messaged me a couple months ago and said “Hey This means that you’re not including commit hashes” and stuff and I’m like news to me, like I didn’t know. Um, I’m using it for legit reasons as part of scripts.

Tim: Yeah, the, it’s actually super messy releasing and signing as well. I had a, an incident, oh, and just GitHub credentials, and just, just git is actually surprisingly  difficult. And  the thing that I didn’t really realize until it became a very large problem, was the fact that git is not immutable. It’s perfectly, git is perfectly happy if you rewrite the history. And  that is, uh, yeah, it makes things like this provenance or like verification of all of the steps really, really difficult if you can’t really, you rely on the git history could just be faked.