I was using an old dev database for work and wanted to use the newer database from production. I deleted my old database and restored to a new database based on the one from production, with the same name as the old database. This made it so that the old database kept being used.
Solution:
I deleted the database using SQL script "Drop database" instead of deleting it from SSMS by right clicking it and choosing "delete". I also had to right click SQL Server from SSMS and choose Properties -> Advanced -> Enabled Contained Databases -> True (instead of false).
For good measure it might be worth it to also delete the .mdf and .ldf files in this folder (depending on your version in the path):
C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA
So if your old databases name is "MyDatabase" you should delete the files "MyDatabase.mdf" and "MyDatabase.ldf".
In order to get access to the folder I had to right click it and give myself permission to enter the folder.