Maidens of the Kaleidoscope

~Bunbunmaru News~ => Front Page Headlines => Topic started by: Alfred F. Jones on February 13, 2020, 08:07:55 AM

Title: The End of Shrinemaiden As We Know It
Post by: Alfred F. Jones on February 13, 2020, 08:07:55 AM
As you may have seen recently (https://www.shrinemaiden.org/forum/index.php/topic,29579.0.html), our head admin, TSO, has died. With her passing, the staff has now had to confront some sobering realities. Not the least of which is that none of us have a login with root access via linode, our service provider. 7HS and Edible have been in contact with linode, but all we can do is pay the site's bills as a stopgap measure, which I have taken care of this month.

However, it's not a matter of paying bills alone. Without root access, the site cannot be maintained; if there is a forum error or disk space issue (which happens regularly), nothing here will be recoverable.

In other words, our days are numbered, and if you have anything on this site you'd like to keep, I would like to advise you to back it up to your own storage as soon as possible.

Attachments will definitely be gone when the forums go under. Artists, please make sure you have everything you want to keep. Writers who have fanfic here, please make sure you have a copy of it; I have several AO3 invitations I would be glad to hand out if anyone needs them for backing up Touhou fanfic or keeping track of authors. Karisa says that HME and the RaNGE/Touhou Projects section have documentation about fan games/patches that might not be posted anywhere else, so if there's anything from those sub-forums you want, grab them now. The old Gensokyo replay sites actually are backed up, at Maribel's (https://maribelhearn.com/gensokyo) and Tom's (http://replays.gensokyou.org/). If you have any friends, members or not, who may be interested in saving something from here, please get in contact with them and let them know.

MotK staff will be updating you on this situation as best as we can, but please keep in mind that this community is currently on a slow-moving train of a site and we know the tracks end somewhere up ahead. We're not sure exactly where the tracks end, but we know it could be sooner rather than later, and everyone should plan accordingly.

Keep an eye on this space for updates. MotK staff may not have all the answers to your questions about where this site is going, but we can at least find out together.

Edit: One of our longtime moderators is working on a replacement forum.  While we won't be able to migrate posts/accounts over there, it should serve as a worthy successor.  A few ancient users here will know this isn't the first time we've moved to a new forum/domain, so hopefully this should be simply another chapter in a very long book.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Edible on February 13, 2020, 01:12:09 PM
So long, and thanks for all the fishgirls.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Kilgamayan on February 13, 2020, 01:29:48 PM
Title: Re: The End of Shrinemaiden As We Know It
Post by: Silent Sinner in Scarlet on February 13, 2020, 04:19:30 PM
I do have an idea regarding this current site's root access, but given this is a public forum, I cannot discuss it here. I can only discuss it in a Discord DM channel.
Title: Re: The End of Shrinemaiden As We Know It
Post by: HakureiSM on February 13, 2020, 06:46:31 PM
Okay, so I guess everyone's seen this coming, and it seems you lot have figured out most of the issues and answered most of the questions.

I have a potentially dumb question, not having interacted with a traditional forum in a few years, and having forgotten how some stuff here works despite my time here: do we have a backup tool of any kind? That is, a way to log and save our posting history, PMs, etc? I have over 11 thousand posts here, and while most of that isn't relevant per se, I wouldn't know how to pick and choose from them all the ones I'd like to keep, the conversations we've had that were meaningful, the fun and sad moments, etc.

Also, I'm pretty glad someone already grabbed the torch and is working on a new platform. This place has always been at the frontline of the western Touhou community, and a spiritual successor is a wonderful idea.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Edible on February 13, 2020, 07:05:27 PM
Nope.  Any preservation will need to be on your end.
Title: Re: The End of Shrinemaiden As We Know It
Post by: FinnKaenbyou on February 13, 2020, 08:08:28 PM
Thanks for the warning. I've grabbed a copy of the DRK thread for safekeeping.

One relevant question - as far as I'm aware, TSO was also the owner of the IRC over on PPIRC. Some of us are still using the s-l channel to hang out. Do we know if anything's going to happen to that?
Title: Re: The End of Shrinemaiden As We Know It
Post by: Edible on February 13, 2020, 09:08:00 PM
PPIRC is owned/operated by others and will remain functional.
Title: Re: The End of Shrinemaiden As We Know It
Post by: HTFCirno2000 on February 13, 2020, 09:27:12 PM
Ok. So the past 2 weeks I have actually been going through the Wayback Machine, and it turns out that this is not the first time something like this has happened to MoTK. There is a lot that i found, apparently the current iteration of shrinemaiden.org seems to go back to April of 2009.

A much better written account of the past than I could ever write.
https://www.shrinemaiden.org/forum/index.php?topic=8592.0

A completely condensed version of what I found in the Wayback Machine:
Originally, in April of 2004 was the first iteration of Maidens of the Kaleidoscope, made by someone named Katchura under the domain shrinemaiden.com. Sometime in either 2006 or 2007 the original site got hacked and the entire database was wiped clean so they had to remake the forum using Simple Machine Forums. Sometime later, shrinemaiden.com expired and the domain parker wanted an exorbitant amount of money for it, that was when shrinemaiden.org was made. Between then and 2009, i'm not entirely clear on the details but sometime around 2009, it was decided to upgrade the website from SMF 1 to SMF 2 so, they had to migrate over to the website we currently have now.

In conclusion, MoTK has this habit of resurrecting itself, like a phoenix. I'm not sure what the admins do actually have planned, but either way MoTK has had a long run, and this is it's longest iteration yet. Kinda a shame it had to happen like this tho  :(

One more thing: I do have this example of what MoTK looked like back in the far past of 2005:
(http://cirnosystems.xyz/motk2005.png)
Title: Re: The End of Shrinemaiden As We Know It
Post by: Karisa on February 13, 2020, 10:36:02 PM
MotK has a long history. I found myself looking into the history on archive.org a year ago, when the site was having its major errors and I thought it wasn't going to be renewed. I've come to consider this version of MotK from 2007 onward, by helvetica and Seventh Holy Scripture, to be MotK v2. It was long before my time, but it shows recreating the site as MotK v3 is doable, as inconvenient as this is. I've become more hopeful since hearing the news two weeks ago.

I wonder if anyone will be able to create a tool that arbitrary members could run to archive their PM inbox and sentbox, perhaps into text files? I've manually saved the HTML of mine, personally, but it was a little tedious to save, and even more so to look back at the archives.

Archiving posts feels like higher priority, though.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Kilgamayan on February 14, 2020, 12:49:54 AM
bofh had a thread crawler once upon a time. Dunno if he would use it again, but it's worth asking.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Sparen on February 14, 2020, 12:58:11 AM
I had a hunch that nobody else had the passwords and it seems I was right, unfortunately.

Is there any form of timeframe we have before the forum collapses? From my own experience, manually backing up and converting even single posts into HTML (with images and links and all) was an excruciating process and it seems this is my last chance to make a complete backup of important RaNGE content.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Aya Reiko on February 14, 2020, 01:08:00 AM
I had a hunch that nobody else had the passwords and it seems I was right, unfortunately.

Is there any form of timeframe we have before the forum collapses? From my own experience, manually backing up and converting even single posts into HTML (with images and links and all) was an excruciating process and it seems this is my last chance to make a complete backup of important RaNGE content.
In addition to that, who owns the domain name?  And about when does it expire?
Title: Re: The End of Shrinemaiden As We Know It
Post by: Silent Sinner in Scarlet on February 14, 2020, 01:48:36 AM
All my core postings on this forum were either music postings referencing my own YouTube & Nico Nico channels or GIF postings referencing my DeviantArt & Pixiv channels (all of them at Alice?s Art Atelier). I have no need to back up such postings since I still own my art/music media on my HDD and on my afore-mentioned creative outlets.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Barrakketh on February 14, 2020, 01:58:47 AM
In addition to that, who owns the domain name?  And about when does it expire?
It looks like it'll expire on April 22. I'd naively assume that TSO owned it, but assuming the whois record isn't BS it claims the registrant is in Ontario, Canada.

From my own experience, manually backing up and converting even single posts into HTML (with images and links and all) was an excruciating process and it seems this is my last chance to make a complete backup of important RaNGE content.
You can copy the generated HTML with the developer tools included with Firefox/Chrome. I just tested it with your first two posts in this topic (https://www.shrinemaiden.org/forum/index.php/topic,13738.0.html) and it worked pretty well.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Karisa on February 14, 2020, 02:00:59 AM
In addition to that, who owns the domain name?  And about when does it expire?
Seventh Holy Scripture, April 22. 7HS is here again, and as the first post mentioned, tried to contact Linode.


Edit:
By the way, you can view the forum on a simpler theme by adding [monospace]?theme=16;[/monospace] to the end of a url. For example:
https://www.shrinemaiden.org/forum/index.php/topic,29584.0.html?theme=16;
To switch back, use [monospace]?theme=14;[/monospace] instead.

It hides signatures, profile sidebars, and other data that's not necessary to the post. It might help if anyone's trying to archive posts automatically.
Title: Re: The End of Shrinemaiden As We Know It
Post by: WishMakers on February 14, 2020, 03:38:19 AM
Some of us in the Land of Codes and Apertures server are getting a scrape of as much content in RaNGE that we can.  Hopefully we can scrape the attachments too.
I want to extend my personal thanks to all the staff who kept the forum alive for so long - it's honestly a miracle that the site has stayed alive as long as it has, especially with the SSL cert stuff.
Here's to the next Touhou haven.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Lebon14 on February 14, 2020, 05:06:18 AM
NOOOOOOOOOOOOOOOOOO

Please find a way to save the website. Please please please please.
Why can't the host be able to make a copy of what's on the server if they are hosting it? Is it encrypted?
Please there must be a way to keep the website alive. Somebody contact archive.org if they could run a crawler to save everything, scripts included.

There MUST be a way to save the forum. The stories... the patches... the role playing... the secret santas... the help provided and solutions found... all... everything... no... a website that I love disappearing is my preservation needs' biggest fear.

Please somebody... find a way to backup the forum database...

At least... somebody... tell me there's a Discord server at least where we'll all be...
Title: Re: The End of Shrinemaiden As We Know It
Post by: HTFCirno2000 on February 14, 2020, 05:17:50 AM
NOOOOOOOOOOOOOOOOOO

Please find a way to save the website. Please please please please.
Why can't the host be able to make a copy of what's on the server if they are hosting it? Is it encrypted?
Please there must be a way to keep the website alive. Somebody contact archive.org if they could run a crawler to save everything, scripts included.

There MUST be a way to save the forum. The stories... the patches... the role playing... the secret santas... the help provided and solutions found... all... everything... no... a website that I love disappearing is my preservation needs' biggest fear.

Please somebody... find a way to backup the forum database...

At least... somebody... tell me there's a Discord server at least where we'll all be...

The forum software is running on a virtual machine hosted in the cloud on a provider called Linode. This virtual machine is like a virtual computer inside another computer. All Linode does is provide said virtual computers and hosts it on their network. Meanwhile, the operating system inside of it is completely self-managed by whoever has actual console access and root login to the OS running inside the virtual machine. If the admins don't have any login credentials to the OS inside the virtual machine, then there isn't a way to actually get to the internal file system to get to the website. Even then, last i remembered reading, the website is hosted using 2 virtual machines, one having the forum software, and the other housing the actual database with all the posts. So it's not just sufficient to get to the server running the website, but also the database behind the website.

Only thing us outsiders can really do is either:
Manually download posts of interest using the browser's built in save function (takes forever)
Setup a script to download the website verbatim (would be a MASSIVE task that could very well exceed hundreds of gigabytes with a large percentage of floof while the actual content would be a smaller portion of that).
Make a web crawler that can parse what and where the actual posts and content are, and store that into a new database. (Hard, would take a lot of coding and algorithms, but theoretically doable)
(EDIT 2/17/2020: Someone ACTUALLY did the last option. Thank you Infy♫ for proving it is actually doable. He has provides a .CSV database of the entire MoTK Corpus later in this thread.)

Either way, it's a monumental task. IF anyone has the hardware, software and time to do it, there's a good chance they might very well archive it.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Lebon14 on February 14, 2020, 05:29:27 AM
The forum software is running on a virtual machine hosted in the cloud on a provider called Linode. This virtual machine is like a virtual computer inside another computer. All Linode does is provide said virtual computers and hosts it on their network. Meanwhile, the operating system inside of it is completely self-managed by whoever has actual console access and root login to the OS running inside the virtual machine. If the admins don't have any login credentials to the OS inside the virtual machine, then there isn't a way to actually get to the internal file system to get to the website. Even then, last i remembered reading, the website is hosted using 2 virtual machines, one having the forum software, and the other housing the actual database with all the posts. So it's not just sufficient to get to the server running the website, but also the database behind the website.

Only thing us outsiders can really do is either:
Manually download posts of interest using the browser's built in save function (takes forever)
Setup a script to download the website verbatim (would be a MASSIVE task that could very well exceed hundreds of gigabytes with a large percentage of floof while the actual content would be a smaller portion of that).
Make a web crawler that can parse what and where the actual posts and content are, and store that into a new database. (Hard, would take a lot of coding and algorithms, but theoretically doable)

Either way, it's a monumental task. IF anyone has the hardware, software and time to do it, there's a good chance they might very well archive it.

I know what's a VM - I just didn't know the website was running using them, let alone 2.
I know that saving a forum is a long and arduous process. I just hope that multiple people will be able to archive what's most important.

Is there an official MotK Discord or will there be one?
Title: Re: The End of Shrinemaiden As We Know It
Post by: PX on February 14, 2020, 05:56:01 AM
Welp, it's been fun. See you on the other side.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Branneg Xy on February 14, 2020, 06:52:31 AM
Ok. So the past 2 weeks I have actually been going through the Wayback Machine, and it turns out that this is not the first time something like this has happened to MoTK. There is a lot that i found, apparently the current iteration of shrinemaiden.org seems to go back to April of 2009.

A much better written account of the past than I could ever write.
https://www.shrinemaiden.org/forum/index.php?topic=8592.0 (https://www.shrinemaiden.org/forum/index.php?topic=8592.0)

A completely condensed version of what I found in the Wayback Machine:
Originally, in April of 2004 was the first iteration of Maidens of the Kaleidoscope, made by someone named Katchura under the domain shrinemaiden.com. Sometime in either 2006 or 2007 the original site got hacked and the entire database was wiped clean so they had to remake the forum using Simple Machine Forums. Sometime later, shrinemaiden.com expired and the domain parker wanted an exorbitant amount of money for it, that was when shrinemaiden.org was made. Between then and 2009, i'm not entirely clear on the details but sometime around 2009, it was decided to upgrade the website from SMF 1 to SMF 2 so, they had to migrate over to the website we currently have now.

In conclusion, MoTK has this habit of resurrecting itself, like a phoenix. I'm not sure what the admins do actually have planned, but either way MoTK has had a long run, and this is it's longest iteration yet. Kinda a shame it had to happen like this tho  :(

One more thing: I do have this example of what MoTK looked like back in the far past of 2005:
(http://cirnosystems.xyz/motk2005.png)




Thanks kindly(and to TSO/helvetica *praying in words&actions* ,7th Holy Scripture,all Mods!Staff,Wordsmiths,Artists and mors) for the Timeline and Hopeful Resume of the "Previous Resurrecting Shrine Maiden Sites" al... There are references and commenting ,dated 2009,alongside  "a few content signs" of a interrupted web databato those mostly-sudden and with data-loss transitions as well among the first Rpg/CyoA Stories in rhe Section of " Rumia's Party Games (very often by " Purvis ")...
Well there is quite some intense and  laborious situation &  engagement  ahead but for helpful starters  there iare already,see beginining post and post starting from previous one on 3/02/2020 at around 20:00 pm( MotK's timestamp date on the Replies within this Thread)some feasible advices,alongside actual copy-download-plans-for- future-reupload-recovery actions or even   (Art n' Music) "Core Posting as References made from External-to-MoTK sites( Youtube,Nicovideo and Deviantart,,Pixiv) being Created Personally Created+Backuppes Data ( the User " Silent Sinner in Scarlet ")   to external-copy from and/or prepare from each own memory drive and then post and or post-by-resume or by-joining..but...!

Title: Re: The End of Shrinemaiden As We Know It
Post by: Branneg Xy on February 14, 2020, 07:05:53 AM



Thanks kindly(and to TSO/helvetica *praying in words&actions* ,7th Holy Scripture,all Mods!Staff,Wordsmiths,Artists and mors) for the Timeline and Hopeful Resume of the "Previous Resurrecting Shrine Maiden Sites" al... There are references and commenting ,dated 2009,alongside  "a few content signs" of a interrupted web databato those mostly-sudden and with data-loss transitions as well among the first Rpg/CyoA Stories in rhe Section of " Rumia's Party Games (very often by " Purvis ")...
Well there is quite some intense and  laborious situation &  engagement  ahead but for helpful starters  there iare already,see beginining post and post starting from previous one on 3/02/2020 at around 20:00 pm( MotK's timestamp date on the Replies within this Thread)some feasible advices,alongside actual copy-download-plans-for- future-reupload-recovery actions or even   (Art n' Music) "Core Posting as References made from External-to-MoTK sites( Youtube,Nicovideo and Deviantart,,Pixiv) being Created Personally Created+Backuppes Data ( the User " Silent Sinner in Scarlet ")   to external-copy from and/or prepare from each own memory drive and then post and or post-by-resume or by-joining..but...!


Not to double-post but about 'Recovery Advices,Planning Actions'' I would like to quote and highlight this offer for backup on https://www.shrinemaiden.org/forum/index.php?topic=23214.new#new (page 14)
 
[[size=78%]quote author=Raikaria link=topic=23214.msg1439682#msg1439682 date=1581351395][/size]
Firstly; I feel like this should be brought up.With the passing of Helvetica (https://www.shrinemaiden.org/forum/index.php/topic,29579.0.html); the main admin of the site, this somewhat throws the future of the forum in the air. [Paying for it; ect]. It's being discussed, but I think it is worth bringing up. I have an idea for a backup plan, but hopefully it doesn't come to that.


 I am going to post over there and PM as well, to help out bridge it out to and from " Raikaria" since he has already warned( on First Post of Rumia Quest Revival)  he needs more time-off between Forum Activity then before : on MotK there are examples of Rumia Quest,Elly Quest,Daiyousei Quest,Wriggle Quest and such).
Title: Re: The End of Shrinemaiden As We Know It
Post by: niektory on February 14, 2020, 08:46:02 AM
Okay, I'm trying to scrape what I can. Let me know if there's anything in particular you'd like me to focus on.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Karisa on February 14, 2020, 10:15:27 AM
Is there an official MotK Discord or will there be one?
Sort of. I've been running Sea of Tranquility (https://discordapp.com/invite/GqJBjX9) (HME-affiliated Discord), which I've seen some people call the MotK Discord. Lately I've been considering trying to expand it beyond the gameplay community into a full MotK Discord, since it already has a starting point.
Title: Re: The End of Shrinemaiden As We Know It
Post by: LunarSpotlight on February 14, 2020, 12:42:10 PM
Hello there, it's been a while since I've been around.  I'll keep things relatively short.

The thing I've enjoyed most about this community at large is the incredible amount of talent and dedication that comes from everyone who is or was ever a part of it.  I'd hate to see this kind of history lost to time.

To combat that, I am running a full sweep of the forums, at least as far as my account can see, using an automated, systematic approach.  It's slow (intentionally delayed to reduce server impact) and so will take time, but I intend to give what I can towards the effort.  At the moment, images and attachments are not covered by my process, so do take care to save what matters to you.  If others are backing some things up, whether selectively or completely, that's probably not a bad thing.  I'll be happy to work with whoever the torch is passed to in order to either help restore or maintain over a decade's worth of irreplaceable history.

- Lunar
Title: Re: The End of Shrinemaiden As We Know It
Post by: Fulisha of Light on February 14, 2020, 12:54:51 PM
It sucks that on my side the site still crashes on desktop due to the expired certificate and Firefox outright ignoring my attempts to bypass it, but I can at least see everything on my phone and save stuff from there thankfully.

I always keep archives of my art nowadays, so there's no issue with the sprites and junk I made and posted here getting wiped. I'll most likely reupload them elsewhere like the new forum or DeviantArt/Tumblr/etc. So to the people that used my stuff don't worry, it'll always be around c:

I'll try to archive some stuff on archive.is and grab a couple of images that I personally want like Dark Kitsune's sprites (which were the ones that inspired me to make my own, but it seems they've been lost due to TinyPic closing down D: damn)

If there ends up being a new forum I hope there'll be a link to it here or something. I'd hate to lose contact with the community here D:
Title: Re: The End of Shrinemaiden As We Know It
Post by: N-Forza on February 14, 2020, 01:12:06 PM
Welp, looks like it's time to make use of that "download PMs" feature at last.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Infy♫ on February 14, 2020, 03:52:27 PM
Welp... Thanks for the good times, everyone.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Karisa on February 14, 2020, 04:49:03 PM
Welp, looks like it's time to make use of that "download PMs" feature at last.
Where is that feature? I can't seem to find it. If there's a built-in method of downloading your PMs, I think it belongs in the first post.

Edit:
It was recently been brought to my attention that there's a Danmakufu Wiki (https://dmf.shrinemaiden.org/wiki/Main_Page) hosted on this site. Has it been archived?
Title: Re: The End of Shrinemaiden As We Know It
Post by: Lebon14 on February 14, 2020, 06:57:01 PM
Can an admin make the login only forums and subforums visible for everybody so we can preserve them?
Thanks.

EDIT:
To be more precise, I'd love to archive, with Wget, the Rules section, Nuclear Furnace, the Caf? and Daiyousei's Cold Storage and all their subforums.
Additionally, I'd love to be able to get the profiles of posters too.

I can do this by injecting my cookies into Wget and get a "logged in" page but that would be I would have to archive all reply/Edit/preview/etc pages for every forums/subforums, threads and posts which slows down the process a whole lot.
Title: Re: The End of Shrinemaiden As We Know It
Post by: DX7.EP on February 14, 2020, 07:55:59 PM
I'm going to hazard that we've until 22 April at latest to do any backups, going by the mention of the domain registration.

If so there will be quite a bit to archive here, mostly old stuff from Genji's Battle Arena where TF2 and Minecraft servers were run and active once upon a time. Some of their contents continue to remain relevant in one spinoff clique.

Aside from that and the 2013 Guilty Alice stuff, I don't know if I'd back up any of my other things.... :V
Title: Re: The End of Shrinemaiden As We Know It
Post by: Agent of the BSoD on February 14, 2020, 08:32:31 PM
This is pretty unfortunate, but given mentions from the staff here that they could only work on so much stuff on the forum, it's not a big surprise. I'll get to work on backing up what is important to me. Also I guess MotK v3 will be a good time to make a fresh guide on the PC-98 emulators.
Title: Re: The End of Shrinemaiden As We Know It
Post by: CapTengu on February 14, 2020, 09:08:01 PM
Godspeed, everyone. I'll miss all of the fanfictions and guides posted here.
Title: Re: The End of Shrinemaiden As We Know It
Post by: HTFCirno2000 on February 14, 2020, 09:34:45 PM
Ok so I did some research, did some digging and actually have created a local instance of Simple Machine Forums 2.0.15. I have found out that, by default the fresh out of the box SMF has database backup enabled. I am unsure if it was disabled at one point but only people who have Forum Administrator rank can do this. It actually allows an administrator to Download the database as an SQL file with a sequence of commands to restore the data on another site. (Also full disclosure, I tracked down the old ModernDark64 theme on the Wayback Machine and used it for the local instance for nostalgia reasons. This is entirely only accessible within my local area network.)

MoTK > Admin Center > Forum Maintenance > Database

The steps are screenshots below:
(https://media.discordapp.net/attachments/571214205257252874/677987352806686740/step1.PNG)
(https://media.discordapp.net/attachments/571214205257252874/677987363430858771/step2.PNG)
(https://media.discordapp.net/attachments/571214205257252874/677987380954923032/step3.PNG)
(https://media.discordapp.net/attachments/571214205257252874/677987136032473121/BACKUP.PNG)

Like i said, i have no idea if it was actually enabled for those who have Administrator role, but if it is there no root access to the server platform is actually required to get the database. However this does not account for the other wikis that exist on this server.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Seventh Holy Scripture on February 14, 2020, 09:36:50 PM
FYI, I have full control over the shrinemaiden.org domain name, and the domain renewal date is irrelevant for the purposes of guessing how long the current forum will remain functional.
Title: Re: The End of Shrinemaiden As We Know It
Post by: CapTengu on February 14, 2020, 09:39:36 PM
Thank goodness. I guess we have until whenever the database decides to call it quits to back everything up.
Title: Re: The End of Shrinemaiden As We Know It
Post by: WindyKitsune on February 14, 2020, 10:05:13 PM
Ok so I did some research, did some digging and actually have created a local instance of Simple Machine Forums 2.0.15. I have found out that, by default the fresh out of the box SMF has database backup enabled. I am unsure if it was disabled at one point but only people who have Forum Administrator rank can do this. It actually allows an administrator to Download the database as an SQL file with a sequence of commands to restore the data on another site. (Also full disclosure, I tracked down the old ModernDark64 theme on the Wayback Machine and used it for the local instance for nostalgia reasons. This is entirely only accessible within my local area network.)

MoTK > Admin Center > Forum Maintenance > Database

The steps are screenshots below:
(https://media.discordapp.net/attachments/571214205257252874/677987352806686740/step1.PNG)
(https://media.discordapp.net/attachments/571214205257252874/677987363430858771/step2.PNG)
(https://media.discordapp.net/attachments/571214205257252874/677987380954923032/step3.PNG)
(https://media.discordapp.net/attachments/571214205257252874/677987136032473121/BACKUP.PNG)

Like i said, i have no idea if it was actually enabled for those who have Administrator role, but if it is there no root access to the server platform is actually required to get the database. However this does not account for the other wikis that exist on this server.

That's very good news, at the end there is a simple way to make a full content backup of the forum.
Title: Re: The End of Shrinemaiden As We Know It
Post by: HakureiSM on February 14, 2020, 10:12:50 PM
Welp, looks like it's time to make use of that "download PMs" feature at last.

I don't see such a feature.

It seems there's a handful of people working on scraper scripts, thank you all for the work.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Kurzov on February 14, 2020, 11:00:05 PM
What's the status of the High Score threads/attached replays, will they be manually moved over or ...?
Title: Re: The End of Shrinemaiden As We Know It
Post by: Kilgamayan on February 14, 2020, 11:01:34 PM
Since forum archive crawlers are being put forth, while I will make no administrative effort to block it, I would personally request any and all crawlers avoid recording LettyJournal, due to the personal nature of many of the threads there.
Title: Re: The End of Shrinemaiden As We Know It
Post by: aListers on February 15, 2020, 12:28:42 AM
Well, I guess it's been a good run. As soon as I heard news of Helvetica passing I backed up my posts so I'm at no loss of history here. My dream diary is safe. That being said, I will be personally backing up my own topics on lettyjournal anyway - they're a good indication of what I thought about my life at the time. I don't want to lose any personal history of mine.

This website has had a huge impact on me as a person. From my dreams of Alice and Gensokyo to the somewhat biased political discussions we had, I've loved my time here. I really needed this place to be able to keep track of the brightest light of my life at a dark time - and while that light may now be put out, I will never forget it and the time I spent chasing it. Thank you to everybody who put up with my creepy and wierd postings on this fourm in my time here. I've cherished every positive message I've recieved on here and wish all the people here the best! Cheers to MoTK! May Touhou and Gensokyo live on forever!
Title: Re: The End of Shrinemaiden As We Know It
Post by: N-Forza on February 15, 2020, 01:20:47 AM
Oh, for some reason I assumed PM archiving was a feature but maybe I was thinking of another forum.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Barrakketh on February 15, 2020, 01:42:01 AM
Oh, for some reason I assumed PM archiving was a feature but maybe I was thinking of another forum.
How important is the stuff in your sent/outbox? I have a "works for me" scraper to backup PMs, but it does require some setup on the users behalf. And it doesn't work on sent messages because I have none to test with :V
Title: Re: The End of Shrinemaiden As We Know It
Post by: Sparen on February 15, 2020, 01:44:21 AM
Where is that feature? I can't seem to find it. If there's a built-in method of downloading your PMs, I think it belongs in the first post.

Edit:
It was recently been brought to my attention that there's a Danmakufu Wiki (https://dmf.shrinemaiden.org/wiki/Main_Page) hosted on this site. Has it been archived?

The Danmakufu Wiki doesn't change much so the entirety of the wiki is available on the Wayback Machine. In addition, since it has already experienced multi-month periods of downtime, I did contingency planning starting a few years back so a good portion of the important parts are already on my website.

Ref: https://sparen.github.io/ph3tutorials/docs.html
Title: Re: The End of Shrinemaiden As We Know It
Post by: Karisa on February 15, 2020, 02:19:01 AM
What's the status of the High Score threads/attached replays, will they be manually moved over or ...?
The attached replays have already been archived manually, and can be found in my scoreboard archive (https://dl.dropbox.com/s/81oxder56pkn8ti/High%20Score%20Entry%202020-2-15.zip). The scoreboards will be recreated on v3, with gensokyo links and MotK v2 attachment links replaced by their archived versions, though it might take a while (I plan to make all the first posts consistent, this time around).
Title: Re: The End of Shrinemaiden As We Know It
Post by: Arcorann on February 15, 2020, 03:47:12 AM
Looks like someone added MotK to ArchiveBot (http://dashboard.at.ninjawedding.org/3), so at least all the public material will be saved.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Evil_Nazgul0616 on February 15, 2020, 06:02:28 AM
I have a question about the status of Rumia's Party Games, particularly the Mystia's Stored Games subforum: The quests archived there tend to span multiple threads of 1000 posts per thread, which makes them a bit difficult and tedious to back up and preserve. I also heard that there were archive crawlers going around through the site, so what I would like to know is if the archive crawlers intend to back up MSG as well or if that would solely be our responsibility?
Title: Re: The End of Shrinemaiden As We Know It
Post by: HTFCirno2000 on February 15, 2020, 06:12:44 AM
I have a question about the status of Rumia's Party Games, particularly the Mystia's Stored Games subforum: The quests archived there tend to span multiple threads of 1000 posts per thread, which makes them a bit difficult and tedious to back up and preserve. I also heard that there were archive crawlers going around through the site, so what I would like to know is if the archive crawlers intend to back up MSG as well or if that would solely be our responsibility?

I'm not sure the specifics of each bot but I think the crawlers are indiscriminate in that they are essentially just archiving anything within the forum structure. So if they are crawling the whole site, they'll get to mystia's stored games too.
Title: Re: The End of Shrinemaiden As We Know It
Post by: N-Forza on February 15, 2020, 06:22:23 AM
How important is the stuff in your sent/outbox? I have a "works for me" scraper to backup PMs, but it does require some setup on the users behalf. And it doesn't work on sent messages because I have none to test with :V
Kind of important because it's how I've been keeping track of orders/requests (although I shouldn't be, admittedly) but most of it's already settled so it's probably the more recent stuff that's actually necessary to have on hand. I'll let you know if I need to make use of it, thanks.
Title: Re: The End of Shrinemaiden As We Know It
Post by: niektory on February 15, 2020, 06:54:50 AM
How important is the stuff in your sent/outbox? I have a "works for me" scraper to backup PMs, but it does require some setup on the users behalf. And it doesn't work on sent messages because I have none to test with :V
I do have some so perhaps I could help with that, just send me the code. But wait... then you'll have a sent message so you won't need my help anymore. ;)
I have a question about the status of Rumia's Party Games, particularly the Mystia's Stored Games subforum: The quests archived there tend to span multiple threads of 1000 posts per thread, which makes them a bit difficult and tedious to back up and preserve. I also heard that there were archive crawlers going around through the site, so what I would like to know is if the archive crawlers intend to back up MSG as well or if that would solely be our responsibility?
I'm backing up all threads I can access with my account, so yes, that includes MSG.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Helepolis on February 15, 2020, 07:55:04 AM
Late to the thread because of time zones and IRL.

RikaNitori contains quite some threads which need to be archived or moved. The wiki is definitely something that needs to be mirrored, but Sparen seems to already have taken care of that.

I will see if I can backup stuff from the sticky threads. The number of danmakufu activity did decline a lot over the past years. I can somewhat see the same happening over at the Eastern Communities so it is not just us.


@ Forza,  right I somewhat promised you to build that app for easier order tracking and such in the past. . . orz
Title: Re: The End of Shrinemaiden As We Know It
Post by: Nemo★Ma on February 15, 2020, 07:31:37 PM
... ...

Just give me a call (via the forum PM, or discord (nemoma#0001) )if you guys need any help on moving the website (server choices, domain name related stuff -but that seems to be taken care of, backup/restore the forum, etc.).

From my experiences, while moving a site like MotK is a tedious process due to its size, it's certainly doable depending on how much one can backup.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Lebon14 on February 15, 2020, 10:56:22 PM
@admins
https://old.reddit.com/r/Archiveteam/comments/f44mxj/touhou_forum_maidens_of_the_kaleidoscope_is_going/fhqfkia/
Quote
Hey u/ajshell1,

I work at Linode, and I'm hoping you can deliver a message for me to the operators of shrinemaiden.org's forum so that we can prevent the site from going down at all. I tried to register on the forum, but it appears registration is currently disabled.

First and foremost, I'm sorry for your loss. From the bit of reading I did, it sounds like TSO / Helvetica was a great friend who was a tremendous benefit to the Touhou community.

Through context from your forum I'm lead to believe that one of the administrators has access to a Linode user account, but they are having difficulty accessing the server via SSH because they do not have the root password. If they have permissions, they should be able to reset the root password (https://www.linode.com/docs/quick-answers/linode-platform/reset-the-root-password-on-your-linode/) of the server, and from there they can do anything they need in order to keep you up and running.

Thanks for forwarding this along if you can.

-Brian
Title: Re: The End of Shrinemaiden As We Know It
Post by: Alfred F. Jones on February 16, 2020, 07:50:00 AM
@admins
https://old.reddit.com/r/Archiveteam/comments/f44mxj/touhou_forum_maidens_of_the_kaleidoscope_is_going/fhqfkia/
First off, happy birthday (to me). (This might be my last birthday on this site and I'm not sure we'll even get to the site's 11th birthday at this rate.)

Second, thank you for passing that on. Unfortunately, it doesn't work. The linode login that I used can only be used to pay the bill and update the payment information used for the site and that is it. It can't even be used to open a ticket with linode, much less reset the root password.

I'm sorry. I wish I had good news with which to reply. Please thank that person for taking the time to try and help, at least.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Edible on February 16, 2020, 08:15:41 PM
Yeah, if we had full access to the linode account it wouldn't be an issue.  Unfortunately that's not the case.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Infy♫ on February 17, 2020, 10:01:56 PM
I spent the past few days scraping the site.
Here is every post on shrinemaiden.org that's accessible through an account. (https://www.mediafire.com/file/ek17fn6nk3r7p3q/shrinemaiden.zip/file)

it's all in a .csv file of about 800mb. I hope someone else can figure out a way to make it all easy to access.
Title: Re: The End of Shrinemaiden As We Know It
Post by: HakureiSM on February 17, 2020, 10:19:11 PM
127mb compressed

Thanks my man, I got a copy
Title: Re: The End of Shrinemaiden As We Know It
Post by: nav' on February 17, 2020, 10:42:26 PM
I spent the past few days scraping the site.
Here is every post on shrinemaiden.org that's accessible through an account. (https://www.mediafire.com/file/ek17fn6nk3r7p3q/shrinemaiden.zip/file)

it's all in a .csv file of about 800mb. I hope someone else can figure out a way to make it all easy to access.
Grabbing a copy too. Myself I have downloaded every thread I could access in print mode, which in total is of similar size when compressed with 7zip Ultra setting. Lightweight, but not so great because links to other posts and attachments don't work... Gotta put together something better.

Edit: I sort of wish your csv file also preserved the id number of every post.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Barrakketh on February 18, 2020, 01:31:02 AM
Grabbing a copy too. Myself I have downloaded every thread I could access in print mode, which in total is of similar size when compressed with 7zip Ultra setting. Lightweight, but not so great because links to other posts and attachments don't work... Gotta put together something better.

Edit: I sort of wish your csv file also preserved the id number of every post.
I'll probably have finished writing my scraper by tomorrow, the next function is just grabbing the main page to get all the boards to crawl. Each thread is combined into one page, quote anchors are corrected to link to the right post, and attachments are saved (the high score entry board was my first test). Images hosted on shrinemaiden.org and in a post are downloaded (covers the staff badges and such, avatars, emoticons, etc.), and replay links pointing to gensokyo are rewritten to so they go to the lunarcast archive.

It's not a mirror of the board. More of a copy of the important content.
Title: Re: The End of Shrinemaiden As We Know It
Post by: niektory on February 18, 2020, 01:58:24 AM
As for me, I downloaded a copy of all (accessible to my account) thread HTML pages in paginated and printable forms, forum attachments linked in them, on-site images included in them in <img> tags, and some other random on-site files linked in them.

So I think I have everything important, now it's just a matter of hammering it into a more usable form. Should I do that? Just upload the raw data? None of the above? I'm asking because we probably don't need multiple people doing the same work independently.

Also, in case the site dies, you can e-mail me at <my username>@tlen.pl.
Title: Re: The End of Shrinemaiden As We Know It
Post by: HTFCirno2000 on February 18, 2020, 04:01:22 AM
I spent the past few days scraping the site.
Here is every post on shrinemaiden.org that's accessible through an account. (https://www.mediafire.com/file/ek17fn6nk3r7p3q/shrinemaiden.zip/file)

it's all in a .csv file of about 800mb. I hope someone else can figure out a way to make it all easy to access.

Wow.
That csv file is extremely useful. It is the entire MoTK corpus in an easy to parse format, that is amazing.
Well done, I was hoping someone would make a crawler that would construct database like this.  :o :o
Title: Re: The End of Shrinemaiden As We Know It
Post by: LunarSpotlight on February 18, 2020, 09:30:07 AM
Since it looks like at least one scraper has finished and a copy has been made available in csv format, I will speed up my script (which had an intentional time delay and is a little more than half completed atm) and get things done with.

On the topic of omitting topics: assuming the others are working similar to mine, the forum topics are structured in a way that's easy to lookup (topic=12345 in the URL and then a second number which increases by 30 [posts per page] for every new page).  That said, the scraper(s) is/are indiscriminate, and any topics that folks want omitted from the archive would need to know what topic number represents each topic.  Topics are numbered in chronological order (date the topic was created), so a category that holds many topics will likely have the numbers for those topics jump around (they're non-sequential).  This would make it difficult to omit any one section of topics.

With some more analysis, it's possible to re-categorize topics into the section they came from, then pull an entire section of topics, but only after everything's already been retrieved.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Suwako Moriya on February 18, 2020, 01:08:44 PM
I mean, the only people that have any real control over what gets archived are the archivers. (In retrospect, I suppose we could have closed LJ viewership when archival was first suggested, but what's done is done.) Since that .csv is now provided, all I can do at this point is ask people to please try to respect the privacy of those who may have posted personal information in that subforum. I'm thinking in particular of one thread from years ago about potential legal name and career path changes of a former staff member (because not a day goes by where I don't think about that thread, for better or worse <_<), but that's merely one example; I'm sure there are plenty of others I can't recall or didn't read in the first place.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Alcoraiden on February 18, 2020, 04:12:26 PM
Well, I'm glad we were able to pull out a copy of the forum. Dang. End of an era going on here.
Title: Re: The End of Shrinemaiden As We Know It
Post by: HTFCirno2000 on February 19, 2020, 03:39:56 AM
Well. For years, I've always been told that digital storage media can hold the information of several dozen libraries. Always taken that fact for granted, but now it really hits home.
That shrinemaiden.csv is on this flash drive, in front of this Alice fumo. The concentrated text of the entire past 11 years completely fits on something smaller than a fumo with QUITE A LOT of room to spare. 11 years of ups, downs, forum games, fanfiction, banter between 2009-2020.

(https://media.discordapp.net/attachments/635926115764011028/679530537244819462/IMG_20200217_235149238.jpg)

It's mind blowing to say the least.
Title: Re: The End of Shrinemaiden As We Know It
Post by: ふねん1 on February 19, 2020, 05:25:58 AM
I'm afraid I'm not familiar with .csv files, is the one that was posted supposed to be basically every post from this site backed up (minus certain exceptions, as mentioned), or is it much more limited than that? I also don't really know how to use it myself, as Excel isn't opening it right lol. I might sound selfish by saying I'm primarily interested in my own post history, but I have a personal project I'd want to legitimately use that for, so I'm curious if I'd have to take other measures to preserve it beyond that .csv file.
Title: Re: The End of Shrinemaiden As We Know It
Post by: niektory on February 19, 2020, 06:49:44 AM
Tom has already put up a mirror of the public part of the forum at archive.lunarcast.net/shrinemaiden.org (http://archive.lunarcast.net/shrinemaiden.org/forum/index.html).
I'm afraid I'm not familiar with .csv files, is the one that was posted supposed to be basically every post from this site backed up (minus certain exceptions, as mentioned), or is it much more limited than that? I also don't really know how to use it myself, as Excel isn't opening it right lol. I might sound selfish by saying I'm primarily interested in my own post history, but I have a personal project I'd want to legitimately use that for, so I'm curious if I'd have to take other measures to preserve it beyond that .csv file.
The .csv file posted contains the text content of (I assume) every post, its title, thread ID, name of the author, date of posting, and subforum it was in. It's not easy to use as is because of its huge size.

If you tell me exactly what you need I can probably help you.
Title: Re: The End of Shrinemaiden As We Know It
Post by: ふねん1 on February 19, 2020, 07:40:55 AM
Well I can't really describe "exact" things I'd want, it's more having the post history to observe my general thought progression through certain periods of my Touhou career. I might be able to get what I want written down sooner rather than later, who knows. Don't worry about extracting anything right now, though thanks for the offer.

But on that note, I imagine we wouldn't be able to log in to Tom's archived version once the original goes down? In terms of my post history, the stuff I'd want to look up for reference is spread across many forum threads, so going to my post history from my profile would be the best way for me to do that on my own.
Title: Re: The End of Shrinemaiden As We Know It
Post by: nav' on February 19, 2020, 11:56:46 AM
Tom has already put up a mirror of the public part of the forum at archive.lunarcast.net/shrinemaiden.org (http://archive.lunarcast.net/shrinemaiden.org/forum/index.html).The .csv file posted contains the text content of (I assume) every post, its title, thread ID, name of the author, date of posting, and subforum it was in. It's not easy to use as is because of its huge size.

If you tell me exactly what you need I can probably help you.
Unfortunately this one still doesn't seem to work properly.

I'm probably going to prepare a small half-functional archive myself, but it's not going to be much more than a curio.
Title: Re: The End of Shrinemaiden As We Know It
Post by: HTFCirno2000 on February 19, 2020, 02:59:28 PM
Well I can't really describe "exact" things I'd want, it's more having the post history to observe my general thought progression through certain periods of my Touhou career. I might be able to get what I want written down sooner rather than later, who knows. Don't worry about extracting anything right now, though thanks for the offer.

But on that note, I imagine we wouldn't be able to log in to Tom's archived version once the original goes down? In terms of my post history, the stuff I'd want to look up for reference is spread across many forum threads, so going to my post history from my profile would be the best way for me to do that on my own.

From what i can tell, all of your posts are in the CSV file and Tom's archive. The only issue though is your name is Japanese characters, and the CSV doesn't seem to have used UTF-16 in the Author names, rather opting to use U+ denotation to store your name. For reference, I'm pretty sure your name is <U+3075><U+306D><U+3093>1 in the csv file.

Main benefit of having the CSV file with everything in it is that most of the hard work of crawling the website and getting all the posts into an easily convertible format has already been done. All that's left would be to figure out a way to make searching the contents and present it in an easy to use fashion.  The way I'm thinking about going about it is to make a converter that can convert CSV into an SQL database, then let SQL handle the searching once it's been converted over. Theoretically, since topic IDs were preserved in the CSV file, to search the database and link to a specific page in the static HTML archive. Would also make it possible to list all the posts made by a user, or show the entire forum thread on a single page.

Might take a while on my end tho, life is busy.
Title: Re: The End of Shrinemaiden As We Know It
Post by: GMOFucker: Makin' GMOs To Fuck 'Em on February 20, 2020, 01:55:04 AM
hello. i am excited to have any really old bad posts i made be wiped off the internet. dont reply to this saying thats all my posts.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Arcorann on February 20, 2020, 03:13:43 AM
It's too late - how many archives are we up to now? Infy's archive, Tom's archive, ArchiveBot's archive (which will be going onto the Wayback Machine once it's done) makes three public archives so far by my count.
Title: Re: The End of Shrinemaiden As We Know It
Post by: GMOFucker: Makin' GMOs To Fuck 'Em on February 20, 2020, 04:09:33 AM
I don't really have a problem with archival, but I think it's better to have the archives hard to find but circulated among the members, instead of publicly indexed on the wayback machine. That way the people who it's emotionally significant to have it but not the random internet weirdos digging for a person's online history
Title: Re: The End of Shrinemaiden As We Know It
Post by: Lebon14 on February 20, 2020, 05:01:21 AM
I don't really have a problem with archival, but I think it's better to have the archives hard to find but circulated among the members, instead of publicly indexed on the wayback machine. That way the people who it's emotionally significant to have it but not the random internet weirdos digging for a person's online history

Don't worry, LettyJournal, while it'll probably backed up, won't be made available per request.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Sapz on February 20, 2020, 02:56:01 PM
Edit: Never mind, seems to be a bit late for that. :V
Title: Re: The End of Shrinemaiden As We Know It
Post by: Hieda no Mukyu on February 21, 2020, 03:11:19 PM
I had been away for months.

While I knew the community was facing some unspecified major difficulties, I had no idea the situation was anywhere near this grave. Yet I was feeling kind of skeptical regarding the future of the Kaleidoscope forums.

Having skimmed through this thread last night and read many parts of it with thought, I am deeply impressed by the community spirit that has surfaced and the self-sacrificing efforts of several members.

Under similar circumstances, many an online community would have just abandoned the ruins even if they had proper root access to the database. Sure, there would have been individuals who would have saved dozens or even hundreds of threads as mementos. Most likely there would have been some sort of continuation, possibly as a number of different Discord groups or whatever the people would have found convenient enough.

As I told one of my friends last night, no social medium that I have discovered so far can serve me like web discussion forums ? as a cross between a chatting circle, a study group, a publication platform, and a searchable online archive.

While I am sad for what has led us here, I am most grateful to those who have helped salvage, save, and revive the discussion forum incarnation of this community. Whether you have contributed with work, money, ideas, or encouraging words, you have been a part of it.
Title: Re: The End of Shrinemaiden As We Know It
Post by: Teetoku on August 28, 2020, 08:52:07 AM
Shame that this forum's days are numbered, RIP to the owner.
I haven't been here for the past few years, just decided to check this place out and then this happenes.