choosing tools and languages | commits of the week #2
the buzz of the week is even the Anthropic CEO wondering whether their artificial intelligence is a conscious being or not (!) Claude gives itself a 15 to 20% chance of being conscious, apparently.
curious, because neuroscientists already estimate that 95% of our own cognitive activity happens outside of consciousness. how does an unconscious being evaluate the consciousness of its own creation? lol that's the question.
anyway, let's talk about something else we think we have control over: deciding on tools to build software. today, 3 takeaways:
- if a lot is already working, it's not worth rebuilding from scratch
- don't switch tools if the tool isn't the problem
- with so many options and variations, not knowing which one is best is normal
let's go (:
by the way, stack, architecture, tools, technology... everyone in the market talks about the same thing in different ways. if you read the last edition about ubiquitous language, here's another example. [link]
1. git merge: when "everything needs to be rebuilt"
deciding which tools to build your software idea with is an important step, especially when you're not a programmer: which is the case for most business people.
for small and medium businesses, this decision carries more weight because the budget to turn back usually doesn't exist, they choose a stack and live with it. but curiously, what I've seen in recent years is: companies with less capital wanting to switch technology, even when the current solution works...
while big companies, which would have the manpower and resources to optimize, end up dragging out the same inefficient solutions for years. both extremes are costly, but today I'm going to talk about the first one.
Joel Spolsky wrote a classic article about this: in 2000, he told the story of Netscape: they looked at the browser's code, decided it was too ugly, and rewrote everything from scratch. it took three years.
meanwhile, Internet Explorer ate their market. Netscape practically died because of that decision.
come on netscape you could've saved us from this
Spolsky's argument is: old code is ugly, but it works. those workarounds that seem to make no sense are usually fixes for real bugs that someone discovered the hard way.
when you rewrite from scratch, you throw away all that accumulated knowledge and you'll rediscover every single one of those bugs again. of course now with AI the time cost of coding is much lower, but it's still worth the read if you've ever thought about "rebuilding everything from scratch"... I'll leave the link at the end of the post.
__
2. git log: the switch that didn't need to happen
the cherry on top of this topic from my experiences with clients was a tax automation project built entirely in Python: scrapers that browsed city hall portals across several Brazilian municipalities, solved captchas, downloaded invoice XMLs, and communicated via Azure Service Bus.
a bot that was already running, already had clients, already had a modular architecture with a factory pattern for each municipality: login, extraction, parsing, all configurable. and with less than two years of the solution, the company decided to switch everything to Java...
over "mere" internal interpersonal issues within the company. it ended up being a few meetings with a more senior dev presenting an architecture that basically had the same elements as the previous one.
the main argument was that the scrapers would run better in parallel with Java... and yes, Java has native multithreading and a mature ecosystem for concurrency, but Python has asyncio, Celery, Scrapy with async pipelines, multiprocessing.
most data collection solutions in production today run on Python, from simple scrapers to data pipelines at companies like Spotify and Instagram. in other words, that kind of solution should also be more than enough for a medium-sized company.
a scraper's bottleneck is almost never the language: it's I/O, rate limiting on the source, poorly done parsing, lack of error handling. meaning, if they had given the other devs more hours on the original solution or hired more people, everything could have been done in Python with the tools the ecosystem already offers.
__
git notes
besides that, this case left me with another reflection: it's good to remember that learning isn't linear and that the learning curve time to switch solutions is almost never calculated.
and I'm not just talking about money, this rework-heavy task can lead the team to burnout (a.k.a. exhaustion), lower output, lower motivation, less team cohesion... it stays between the lines. most devs enjoy solving problems and learning new things, they'll say yes to change, but there's this whole hidden cost right there.
I chose this little story because we've all seen variations of it, and it's a recurring topic in software development that will become even more common now with the boom of miraculous promises and more efficient solutions with AI...
git diff: two devs, same problem, different answers
in the case of solo devs (as I mentioned in the last post), architecture choices don't need team votes, the responsibility is yours and yours alone, as is dealing with what you picked later, for better or worse lol this week I had proof of how the same problem generates completely different solutions.
I had a meeting with someone who built something similar to what I'm building, being used over in Spain. here are some technical points we saw: we both need to store coordinates in the database, but I use Supabase and he uses PostgreSQL with PostGIS.
since Supabase runs on top of PostgreSQL, in theory I could enable the PostGIS extension and have the best of both worlds: native geospatial queries like "give me all properties within a 5km radius of this point". today I don't do that. I do geographic filtering more on the front end than in the database, which probably won't scale well.
speaking of georeferencing, his solution relies on a more centralized Spanish database, with very little missing data. mine, on the other hand, has tens of thousands of properties, most without latitude and longitude, so I use the Mapbox API to populate that information. so for him, Mapbox is mostly just visualization, whereas for me, it's data infrastructure too.
however, for visualization I use Mapbox GL JS directly on the client side: full control over layers, filters, popups, clusters, all via code. he uses Mapbox Studio to build it, embeds it in the front end, and said that way the user experience is faster and smoother.
and the funny thing is that neither of us "chose" the map approach on purpose. I use Next.js, he uses Angular: Mapbox GL JS integrates naturally with React, so getting hands-on with the code was a pleasant route. in Angular, embedding through Studio avoids a headache... so the stack kind of induced how we used the map.
it's also worth thinking that he picked his stack 3 years ago, I picked mine now... and obviously 3 years from now both of us will look at the solution and think "this and that could be done differently". the list only grows and nobody has time to test everything before deciding or completely changing stacks to keep up with what's modern and more, more optimized.
Netscape rewrote the browser from scratch and lost the market. the company switched from Python to Java with the product already running and probably paid an extra cost for it without strictly needing to. and two devs who never talked to each other, solving the same problem with similar tools, made opposite choices... and both work.
people love endlessly discussing tools, but in these three cases on totally different scales, the problem was never the tool. if you're a dev and you're lost in this sea of options, or if you're in business and don't understand why every dev suggests a different stack:
welcome to the club. there's no 100% right recipe, there's the near-ideal and there's what works for your context right now: you choose with what you know, build, and adjust along the way reasonably. what matters is solving the users' pain points.
__
other branches
links and references that caught my attention this week
_ Joel Spolsky, "Things You Should Never Do, Part I": why rewriting software from scratch is almost always the wrong decision. written in 2000, still relevant. [link]
_ Dario Amodei on NYT's "Interesting Times" podcast about not knowing if Claude is conscious. [link]
_ about the neuroscience reference: the 95% figure comes from cognitive neuroscience studies, it's debated but widely cited. the Max Planck Institute study showed that you can predict a person's choice up to seven seconds before they consciously decide. [link], but I discovered this through Sam Harris's book "Free Will" (2012), which I highly recommend [link]
Until the next commit, Michelle (:
