James Vasile's Blog

Every Noise

2023-05-22T00:00:00-04:00

Every Noise is a website for music discovery. It is spare. You don't experience it with your eyes.

Its main page is a tag map of genres. Scroll or search, click to get a sample, repeat. It is absolutely thrilling fun. It turns out I like Sambass?

Every Noise sifted through the most popular music of 2022 from around the world. They have samples and spotify playlists for a ton of genres, and you can narrow by country. I haven't started exploring beyond the US yet.

You can enter an artist's name and Every Noise will show you all the lists they're on so you can explore similar songs. That's a very cool feature. Unfortunately, many of the artists I initially entered were not popular enough to be on any lists. And the ones who were (Rosalia, Bad Bunny) are too popular for that to be useful. The place this feature sings is with artists like Gorillaz, that genre hop and aren't so popular as to be everywhere.

Combined with the genre map, the hit lists are a great tool. You can find a genre on the map, then do a text search on the 2022 Around The World page, click it and get a bunch of related artists and songs. One thing that's interesting is that the most popular songs of related genres all tend to sound closer to each other than the samples on the genre map do. This is especially noticeable with things like Trop House, where the "tropical" part gets flattened into the vibe of popular dance music that gets music on to hit lists.

And for all that there are a lot of genres here, it is more striking in its breadth than its depth. Any fan of a genre could parse things into microniches and maybe if you're using "sounds like this other artist I know" as your starting point, that would be useful. Still, what makes music good isn't how many adjectives it takes to name its genre. What you really want is music that is ineffably catchy and charming, and that quality will never be reducable to MP3 tags in an itunes library.

I spent way more time on this site than I should. It was like roaming the shelves in a labyrinthian alibrary. It's easy to get lost in the stacks. Below are my tasting notes.

Concentrated Focus

So much of my music listening isn't actually listening. It's more like curating a sonic background for my brain to track while I do other things. My first foray into this list was to look for the lulls.

Ambient Techno is what the inside of my head sounds like when I'm trying to fall asleep at a party.
Compositional Ambient was a delight. It's perfect for when you're doing sunrise yoga in your hexayurt.
The Lounge and Sunset Lounge playlists are pleasant.
The Escape Room is a surprisingly good listen if you want to concentrate with some urgency and you can take background noise. I guess that's what it's like in an escape room?
Their LowFi Beats are not low fi, but they are slow beats without the crackle and pop. Chillhop is also pretty on point. If you're in this mode, you might also enjoy Focus Beats, which clocks in at a slightly higher bpm. Similarly, Focus and Neoclassical are pleasant if you trend acoustic. Meanwhile, the LoFi Chill playlist steps it up a pace but is vocal-forward. A more melodic alternative is the Chill Wave playlist.
Vapor Soul is nice and chill without being super slow.

Jazzy

The Electric Blues playlist is not electro or electronic, just electric, which is quaint buy rhythmically satsifying.
The Jazz Funk playlist is funkier tho.
Jazz Piano is every hotel lobby with a live pianist you've ever been to.

House and Electronic

Turntablism does not feature turntablism, but if you like the flow of Method Man and KRSONE, you will enjoy it.
Minimal Synth is exactly what it says: just people playing with synths. It's like 90s tracker mods.
And of course I had to listen to the Deep House playlist, which is... poppy? Full of vocals, not minimal at all. Why have a Vocal House category if everybody is cramming divas and baritone drones into their house? And the Bass House is all over the place. It even gets a little jungle at times. My quest for deep, minimal house that doesn't devolve into tech house continues.
If you have to do tech house, at least make it funky.
I am secretly into Witch House, but the Russian Witch House is way extra.
The Electronic Trap playlist has too much of the wubs.
Are you into Progressive House? I wasn't, then I listened to the most popular tunes of 2022 and... I'm nostalgic for progressive.

Everything Else

I refused to click on Meme Rap. There is good hip hop out there. I guess I should dig into Every Noise and find it.

Installing Matrix's Synapse!

2023-05-20T00:00:00-04:00

We are going to follow the standard instructions for installing Synapse.

Install and Config

$ sudo apt install matrix-synapse

I chose a servername of jamesvasile.com and enabled anon stat reporting.

Reverse Proxy And Delegation

We want to share this box with other services. The root domain is doing other things, but we want people to be able to get to us via jamesvasile.com, not matrix.jamesvasile.com. So we use delegation. We provide a pointer at https://jamesvasile.com/.well-known/matrix/server. The pointer is just a JSON file that tells clients where to go for the real server.

Delegation

Server

We already serve a site at jamesvasile.com so we'll add .well-known to it. That site is a Pelican static site, so this is easy.

Frist we add .well-known as a static path in pelicanconf. Then:

$ mkdir -p content/.well-known/matrix
$ echo '{ "m.server": "matrix.jamesvasile.com:443" }' > content/.well-known/matrix/server
$ push # send all this to our live server

Then we check it with:

$ curl https://jamesvasile.com/.well-known/matrix/server

and see if we get our JSON snippet back. If we do, then it's working.

Client

Now we should do the same for clients:

$ echo '{"m.homeserver": { "base_url": "https://matrix.jamesvasile.com" }}' > content/.well-known/matrix/client
$ push # send all this to our live server

As above, we can check it with:

$ curl https://jamesvasile.com/.well-known/matrix/client

and see if we get our JSON snippet back. If we do, then it's working.

Reverse Proxy

We are serving our static site with Caddy, so we just need to configure it. We configure Caddy by editing /etc/caddy/Caddyfile to include two new stanzas:

matrix.jamesvasile.com {
  reverse_proxy /_matrix/* http://localhost:8008
  reverse_proxy /_synapse/client/* http://localhost:8008
}
jamesvasile.com:8448 {
  reverse_proxy http://localhost:8008
}

Then sudo service caddy reload

Postgres

We installed via apt-get, but that doesn't include postgres, so let's apt-get it.

# apt install postgresql

From there, we'll follow the official docs.

# su - postgres # change to postgres user
# createuser --pwprompt synapse_user
# createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse

Then adjust the database part of /etc/matrix-synapse/homeserver.yaml:

database:
  # The database engine name
  name: "psycopg2"
  args:
    user: synapse_user
    password: <PUT PASSWORD HERE from /box/jamesvasile.com/synapse>
    database: synapse
    host: localhost
    cp_min: 5
    cp_max: 10

Other Synapse Config

In /etc/matrix-synapse/homeserver.yaml:

suppress_key_server_warning: true
macaroon_secret_key: <PUT PASSWORD HERE from /box/jamesvasile.com/synapse>
no_tls: true

Troubleshooting

If sudo service matrix-synapse restart doesn't get you a running server, you might want to run it manually so you can interact with finer control and see the output live.

# cd /var/lib/matrix-synapse
# /usr/bin/python3 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/

Running this showed me I was missing the macaroon key, for example.

Add User Account

At this point, you should have a running synapse instance.

Registrations via client are closed to prevent strangers from using the server. I'm not sure how it works. Maybe it's possible that with the registration_shared_key, a remote user could register, but I used the commandline to register myself as a user.

Note that the usual instructions say to use register_new_materix_user but the Debian package prefixes that bin with synapse.

$ synapse_register_new_matrix_user -u james -a -c /etc/matrix-synapse/homeserver.yaml

Try it out!

I fired up nheko and it instantly complained about keys despite my not having given it any login info. Perhaps when I tried (and failed) to login earlier it left some stale data. I deleted ~/.config/nheko/nheko.conf and restarted. It was fine after that.

Matrix ID:  @james:jamesvasile.com
Homeserver address: https://jamesvasile.com:8448

If you have to add :8448 after your homeserver address to make it work, the client delegation isn't working. Check the client delegation file and the caddy proxy.

UPDATE

Well, it runs and I can connect a client to it, but... I tried to join a large room or two and it ate 100% CPU and I will never do that again.

It seems like matrix is good for small, unfederated instances or maybe federation if your users never join any large rooms.

And now the server is borked because those large room joins are in the database. Any time I start it up, I synapse tries to sync up with those rooms and eats the CPU again. I can see the rooms in the db:

SELECT room_id FROM current_state_events
WHERE state_key = '@james:jamesvasile.com'
AND type = 'm.room.member'
AND membership = 'join';

I tried to use synadm to delete those room memberships from the commandline, but it doesn't support that.

I contemplated just removing those rows from the db, but I don't know what that would break.

I guess I could try to figure out how to construct a curl request to delete the room joins via the admin api, but I am out of time on this project.

I could just blow away the database and start over, but I came across some ominous warning that then the rooms I tried to join would forever bar me or something like that. That seems... brittle? Like maybe decentralized, self-hosted systems should assume sometimes people are going to experiment or do a fresh install?

Maybe I need to migrate synapse to another box.

The funniest part about all this is that I keep seeing messages to ask for help in the synapse matrix room, which I cannot reach because my synapse server isn't working.

Ah well, exploring matrix will have to wait for another day!

Open Sourcing Checklist

2023-05-19T00:00:00-04:00

Taking an existing, non-open codebase into the open is one of those tasks that seems relatively simple --- apply open source license and publish the code. In practice, however, there is some hidden complexity along the way, and many projects start out with the best of intentions but fail to complete the process.

This, of course, is why we always advise people to be open from day one.

This document is a checklist of steps you might take before you finally publish your work under open source license. Most steps are optional, but all of them have value. Think of them as best practices, not mandatory requirements. Consider each step and whether it applies to your project. Then narrow the checklist to the steps you will take. Feel free to do these in any order that makes sense to you. Good luck, and please ping OTS at info AT opentechstrategies.com if we can help.

This document is not an exhaustive list of concerns you should consider when open sourcing. It doesn't go into the reasons why you are considering open source (and we have found that going open without a clear goal in mind usually fails to achieve any goals at all). It doesn't include building a community or a sustainability model (open source is not, in and of itself, a business model). All of those are considerations of a much bigger picture than this checklist contemplates.

[ ] Get internal buy-in. Taking code into the open can be a big deal. It can disturb existing strategies or revenue streams. Often, publishing code is the right answer overall, but might impose tradeoffs that impact some stakeholders more than others. Some of those stakeholders might become net losers. Make sure you start the process of obtaining the necessary buy-in from internal stakeholders early. Secure support all the way up the org-chart if you can. This is the step that at first seems doable or even easy but ultimately blocks so many projects from ever going open.

[ ] Get legal buy-in. Let your counsel know you're moving in this direction. Make sure you know what their requirements are to sign off on publishing under an open source license.

[ ] Ensure regulatory compliance. Check to make sure you're square with export control regulations regarding encryption. If you're just using commonly-used open source encryption libraries, you are probably fine. Depending on what your software does, other regulatory rules might apply. Find out and get in compliance.

[ ] Check your copyrights. Ensure you have the needed permissions to release your project under open license. These permissions could come from licenses or copyright ownership. Do not assume you have the rights. Go look. Do an audit. If necessary, read up on works for hire and federal ownership of copyright.

[ ] Consider your patent portfolio. FOSS licenses sometimes include patent permissions, whether explicitly stated or implicitly granted. If you or your contributors hold patents, you should evaluate the impact of open licensing on your patent rights. You might decide your patents are best used to nurture your open source strategy. Alternatively, you might want to take steps to protect your patents.

[ ] Pick an outbound license. Usually this will be BSD, Apache, or some flavor of GPL, but there is a wide range of FOSS licenses out there for you to choose from.

[ ] Consider an inbound license. Sometimes these licenses are called "contributor license agreements". By default, a project usually takes code under an inbound license that is equal to their outbound license. If that outbound license gives the project all the rights it needs for the foreseeable future, that's good enough. If it doesn't, you might consider a contributor license agreement. Project Harmony has some sample agreements that are old but worth reading (full disclosure: I worked on that project).

[ ] Ensure license compatibility with dependencies. Two pieces of software are license compatible when they can be combined into one work and released together without violating the licenses of either of the original pieces of software. The combined work is then treated as goverend by only one of those two licenses, usually the less permissive license.

[ ] Upload your code to an online repository. This will probably be a git repository hosted on GitHub, LibreHQ, or a GitLab instance. Do not make it public it yet.

[ ] Add a README.md file. Place it in the root directory of your repository. It should specify the name of the software and describe what it does in lay terms. Include contact information, directions to your mailing list or Zulip/Slack /Discord instance, and your general level of willingness to accept public participation. Provide basic instructions for installation or refer readers to online documentation.

[ ] Test installation. Give repo access to a new user or developer who has never installed your software before. Have them follow your instructions. If they got a running installation, good. If not, fix your instructions before you publish.

[ ] Fix your test suite. Ensure your tests pass. Sideline (e.g. mark "expected fail") those that don't. New users will often run tests to make sure their install worked. If the tests fail, it will be difficult for them to know whether that is because the install failed or because that test stopped passing and nobody cared.

[ ] Add a LICENSE or COPYING file.

[ ] Add a CONTRIBUTING file. This file tells people how to contribute.

[ ] Prepare for public participation. Pay particular attention to your planned cadence of replies to incoming interest. Your approach to inbound messages will, of course, depends on your community strategy, which is beyond the scope of this checklist (see our Archetypes Analysis and Producing OSS for starting points). Document that approach in your README or CONTRIBUTING file so people can form expectations about how quickly you will respond to issues and PRs. See

[ ] Add issue templates. These can help guide public participation. Give people some hints about where discussion of issues happens. Typically, public projects default to the issue tracker, but some prefer the project's mailing list or forum.

[ ] Examine ownership of your repository. Make sure multiple people have authority and actual ability to administer the repository.

[ ] Squash history. Does your repository contain sensitive information or secrets? Has it ever? When repositories are private, people tend to be less guarded about what they check in to the repo. Problematic content includes secrets and PII but also might include commit messages containing profanity or even defamatory content. It is not practical for most projects to examine all their history, so many delete, edit, or squash history before going public.

[ ] Remove secrets. Ensure there are no API keys or passwords or other sensitive information in the remaining codebase.

[ ] Publish documentation. Centralize it into an accessible location under open source or creative commons license. Documentation should be open to change requests. Track down the design and usage documentation that may currently reside in various internal places and make them public. Collect the implicit learnings of your team, and write them down so people without your experience can productively approach the software.

Remember, this checklist is a general list of tasks, some of which might not apply to you. Adapt it to your specific needs and legal requirements. Work with your lawyer and your open source advisor to reduce this to the list that works for you and your product.

Thanks to Karl Fogel, Frank Duncan, and Jesse Bickel for reading early drafts of this checklist.

Main screen turn on!

2023-04-13T00:00:00-04:00

All this self-hosted fediverse stuff has me nostalgic for the blogs of yore.

I also needed a place to put some notes where I could share them.

You might notice that this post is not the oldest item on the site. I am backfilling some of the other things I've written from other places.

Spot The Pattern: Commoditization

2019-10-02T15:28:00-04:00

(This is the fifth post in our Open Source At Large series.)

Last week, I spoke with the CEO of a company that makes a proprietary, category-leading workflow product. He asked me "what should we open source and when?"

This is one of the most common questions we get. There is no easy, standard answer to this question, but I usually start by looking for features of the ecosystem that either are or could be commoditized.

A commodity is something that is the same no matter who makes it. Gasoline is a commodity. Customers don't really care where they buy gas. They want it cheap and reliable, but the gas itself should be the same stuff regardless of which oil company refines it and pumps it into your car.

In practice, of course, gas from different companies is not identical. Some places enhance it with detergents, mix it with ethanol, or add stabilizers. But for the most part, consumers ignore these small differences. You put it in your car and it works. People pick gas stations based on price and convenience.

It's hard to charge a premium for a commoditized good. People are not willing to pay extra for a product if they can get the equivalent for less elsewhere. As a result, gas stations compete on price. This squeezes margins, and so in the end they make their money on the convenience store, just like movie theaters do with popcorn.

Open source software businesses are often in the same position. There's a core offering that is, to put it bluntly, boring. Anybody could offer this boring core without innovating. You cannot be in the market without matching the core's offerings, and customers want standard interfaces these days, so there's not much opportunity to differentiate on the basic core offering.

Database servers follow this model. From a business perspective, storing rows of data is not interesting. Everybody can do it, and nobody really does it much better than anybody else. There are performance improvements at the margins, and we can build interesting services on top of storing information, but basic data storage and retrieval is just facilities maintenance, not the thing your customers choose you for.

Of course, not all data storage is basic. You can store things faster or in ways that make access easier. You can stack data vertically instead of horizontally. You can keep all the data in memory, duplicate it across multiple servers, or sync with client-side storage. These are features that can set one product apart from another. You might be able to charge a premium for them. None of them, though, is actually the boring, basic task of putting data on a disk and making sure you can find it later.

For a business that manages data but can't find competitive advantage on storing it, the database is a cost. It's not worth spending a lot of money to develop or acquire a database that improves on the standard set of features because there's just not much improvement to be had. Best to source those features as cheaply as possible.

Of course, the same logic applies to everybody else in your industry. We're all trying to get the boring stuff done as cheaply as possible. This is where open source cooperation comes in. We can't compete on those features, so there's no point being competitive about it. We use open source to get together and collaborate with all the other folks who want rock-solid databases. Open source lets us do this even if we're in competition on a range of other fronts.

We see this pattern a lot in free and open source software. Product categories go open source, then contract, then perhaps expand when somebody figures out the next frontier of differentiation. Most recently, we've seen this effect around web rendering engines. Every browser's goal is to depict HTML in standard ways. Aside from being at Google's mercy, there's little reason for most companies to make their own rendering engine when there's a pretty good one that is open for the taking.

Spotting this pattern early is a crucial step in knowing what to open source and when. You can lead or let somebody else lead -- either might be strategically useful, but spotting the pattern lets you choose which position to take.

Sometimes it makes sense to open source what you have mainly as a signal to others that an area is becoming commoditized and so they should open source their stuff too. That is, you don't necessarily have to be aiming for a dominant position in a new open source area -- in fact, you often don't have to decide that in advance. Instead, you open source because an area is about to become commoditized, and the earlier you shift your investments, the better positioned you will be get the type of influence that serves your needs in the inevitable post-commoditization universe. (For more about the various forms that influence can take, see Open Source Archetypes.)

Once you start looking for the commoditization pattern, you see it all over the open source world. It is a common tool for understanding the strategic position of all kinds of products. In addition to the basic pattern, there are two more big concepts around commoditization worth considering: the cyclical nature of commoditization and the factors that allow resisting that cycle. We'll cover those in future posts.

Thanks to Microsoft for sponsoring the Open Source At Large blog series.