⏳ Archiving the old General Discussion forum

Arete_Hime - Sorry, I haven’t made a contribution to this task yet. I did spend some time trawling through the old forum pages over the weekend but quickly found I was losing the will to go on living. There was too big a risk that I was simply going over the same ground that you and @cos had already covered.

Anyway, it helps that the forum search tool on the Memrise Users Wiki site still works. That, at least, helps speed things up…if you can remember what it is you are actually looking for.

The short descriptors you have used in your last list will be a big help too!

Thanks for all your hard work.

Arete_Hime · May 11, 2016, 11:23am

I’ve now trawled through the first 50 pages of General Discussion and generally saved maybe 1-4 threads per page. As I went on I included more of the first post (I’ll go back over the first links as well and include some text). I’ll do Course Ideas probably as well over the next couple of days.

I do find myself wondering what the use of it is. We collectively know enough tips and tricks to not need to save those. And as the discussions get older they become less applicable and interesting. The two kinds of posts that keep being applicable and useful I think are userscripts and recommendations of courses.

As per usual, I’ve written an Autohotkey script to speed things up, my procedure:

Prerequisites: one open instance of Notepad (probably called links.txt) and Chrome, Autohotkey script active, Chrome extension from my first post installed.

Have a Chrome tab open on the General Discussion/Course Ideas forum, middle click any threads on it that look interesting (to open them in a new tab) until you’ve collected a few. Right click inside the threads you want to save and click “Save this page to the Internet Archive”, do something else until that’s finished. Highlight the text from the post you want to include to describe it and hit F9 (this pastes the text to Notepad and then pastes the url to Notepad and takes you to the next open tab in Chrome).

#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn  ; Enable warnings to assist with detecting common errors.
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.


SetTitleMatchMode 2




F9::
{
Send, {CTRLDOWN}c{CTRLUP}
WinWait, links.txt - Notepad, 
IfWinNotActive, links.txt - Notepad, , WinActivate, links.txt - Notepad, 
WinWaitActive, links.txt - Notepad, 
Sleep, 100
Send, {CTRLDOWN}{END}{CTRLUP}{ENTER}{CTRLDOWN}v{CTRLUP}
Sleep, 300
WinWait, Memrise - Google Chrome, 
IfWinNotActive, Memrise - Google Chrome, , WinActivate, Memrise - Google Chrome, 
WinWaitActive, Memrise - Google Chrome, 
Sleep, 400
Send, {CTRLDOWN}l{CTRLUP}
Sleep, 200
Send, {CTRLDOWN}c{CTRLUP}
WinWait, links.txt - Notepad, 
IfWinNotActive, links.txt - Notepad, , WinActivate, links.txt - Notepad, 
WinWaitActive, links.txt - Notepad, 
Sleep, 100
Send, {CTRLDOWN}{END}{CTRLUP}{ENTER}{CTRLDOWN}v{CTRLUP}{ENTER}{ENTER}
Sleep, 300
WinWait, Memrise - Google Chrome, 
IfWinNotActive, Memrise - Google Chrome, , WinActivate, Memrise - Google Chrome, 
WinWaitActive, Memrise - Google Chrome, 
Sleep, 300
Send, {ALT}
Sleep, 500
Send, {CTRLDOWN}{TAB}{CTRLUP}
}
return

Arete_Hime · May 11, 2016, 12:19pm

… by the way, my internet is slow so everything takes 5 times too long. If you or anyone wants to help, please do.

cos · May 11, 2016, 3:01pm

Part of the point of this is that when people run across links to memrise forum threads and those links lead to a 404, they may think to try archive.org, so it’d be good if linked-to posts were there. I’ve been going through my memrise notification emails and archiving those.

DrewSSP · May 12, 2016, 12:47pm

I found this tool that looks like it could really speed this whole process up: https://internetarchive.readthedocs.io

It’s apparently a command line interface that allows one to interact with archive.org. If somebody got a list of links to each of the forum posts (probably not difficult to get) this could potentially be used to upload them all with one simple command.

My biggest concern is how easily searchable any of the information would be.

DW7 · May 12, 2016, 12:57pm

Many thanks Arete_Hime, for doing this valuable but onerous job,

but not wishing to sound ungrateful, I wonder if when someone visits a link they could paste the title of the thread above the link - otherwise we could end up looking for ages for a specific post we are interested in, unless ‘we’ had actually made a note of the thread number (which I’m afraid I never did - only the title).

Arete_Hime · May 12, 2016, 3:21pm

Thanks. I looked at it. It looks like that only covers uploading (items that are on your computer) and not archiving URLs? If it can do bulk archiving of URLs, I agree collecting them wouldn’t be a major issue.

@DW7, I did that later and I will go back over the ones for which I did not do that.

DW7 · May 12, 2016, 3:31pm

Thank you so much Arete_Hime - I thought that when I scrolled down.

It will be so helpful

CertifiedDoc · May 16, 2016, 5:43pm

Hey, this is a great idea. I’ve got a few things to say that I hope will help.

First, I’d like to point out that archive.org does indeed make copies of images, so we shouldn’t be worried about losing them. If you look at the URLs for images in the page source, they all have /web/ before them. If this wasn’t the case before, I suspect that it takes archive.org a while to download all the images from the original page to its servers.

Second, according to the documentation that @DrewSSP provided, it’s true that the Internet Archive API can only be used to upload documents from your local machine. However, there is a workaround for this if you access to either CygWin or a Linux machine.

You can pipe the results of a curl command into ia upload, as shown in the documentation here: https://internetarchive.readthedocs.io/en/latest/cli.html

However, there are two caveats to this:
First, curl downloads the html from a webpage to your machine. Essentially, this means have to wait for every page to download before it can be uploaded back to archive.org. It’s automated so you don’t have to watch it, but it could be time consuming.
Second, the upload command only works for one page at a time. You’d have to get the URLs beforehand, store them in a file, and write a script to run ia upload on each of them in turn.

There’s one more possibility: Scrapy. http://doc.scrapy.org/en/0.14/index.html#
Scrapy is a PHP API designed for building web crawlers. This would allow us to create a PHP program to crawl the Memrise forum and pick out the URLs for each post. Once aggregated, the PHP script could be coded to upload each post to archive.org on its own.

Furthermore, you stated that your hope is to paste the threads here. A web crawler could be used to reformat output from each original post automatically. From there, it would be simple to paste each post here.

You could skip archive.org and build the posts immediately with Scrapy, but I’m not yet sure if there is a way to automate the upload process to this site. You might run into spam blockers trying to do that.

Arete_Hime · May 17, 2016, 11:04am

Thanks. I’ve exceeded the time I’m willing to devote to this already, and I don’t know how to write the scripts required, so I am not going to do this.

I have now trawled through 50/230 pages of the General Discussion forum and archived 203 threads. It’s unlikely I’ll do anymore from that.

I’ve trawled through 26/42 threads in Course Ideas and archived 196 threads. I will perhaps try to finish that, or maybe not, can a 2-year-old thread still have useful info?

As the threads get older, it becomes close to impossible for me to determine if they have any useful content without reading them (which I am not going to do) as I have read none of them.

Oh, and then there are the other forums, like http://www.memrise.com/forum/3/ Feature Suggestions

If anyone wants to continue…

pdao · May 17, 2016, 12:30pm

@arete_Hime ~ I have to confess that I don’t understand a great deal of what everyone is talking about with this thread, but I just wanted to say “thank you” for all your efforts on behalf of the Memrise community. You went above and beyond, and stepped up when no one else did. I commend you for that, and I am sure others do as well. Thank you. 謝謝 !

Arete_Hime · May 17, 2016, 4:48pm

Thanks I figured it would take me a few hours, so why not. It took a bit longer though. The only bother is that my internet is slow, making it take longer than it should.

Arete_Hime · May 30, 2016, 2:31am

If the threads are short and don’t contain images, like most threads in the course forums, not archiving the topic on archive.org and just copying and pasting to Notepad is a lot quicker. So here’s a guide to do that very quickly. The end result if you were to paste it into a post here would be every post looking like a quote.

Go to https://autohotkey.com/download/ and install it.

Create a Notepad file and call it anything you want. I suggest Autohotkey copying posts.txt paste the below into that:

#NoEnv  ; Recommended for performance and compatibility with future AutoHotkey releases.
; #Warn  ; Enable warnings to assist with detecting common errors.
SendMode Input  ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir %A_ScriptDir%  ; Ensures a consistent starting directory.

SetTitleMatchMode 2

F9::
{
Send, {CTRLDOWN}c{CTRLUP}
WinWait, posts.txt, 
IfWinNotActive, posts.txt, , WinActivate, posts.txt, 
WinWaitActive, posts.txt, 
Sleep, 100
Send, {CTRLDOWN}{END}{CTRLUP}{ENTER}{ENTER}{ENTER}[QUOTE]{ENTER}{CTRLDOWN}v{CTRLUP}{ENTER}[/QUOTE]
Sleep, 300
WinWait, Memrise - Google Chrome, 
IfWinNotActive, Memrise - Google Chrome, , WinActivate, Memrise - Google Chrome, 
WinWaitActive, Memrise - Google Chrome, 
}
return

Save and close it and rename it to: Autohotkey copying posts.ahk

Double-click it to make it active. You should now see a small green “H” icon in the lower-right corner of your screen.

Create another new Notepad file called posts.txt and just keep it open. The posts will get copied to this.

Open Chrome and open any forum threads you want to save.

The Autohotkey script I wrote does the following: when you press F9 it copies anything you have highlighted, goes to post.txt and pastes it there (it also puts [QUOTE][/QUOTE] around the pasted content), it then comes back to Chrome.

So in Chrome, highlight anything you want saved and press F9.

Rabolde · June 7, 2016, 4:24pm

Which Autohotkey version do you suggest?

Arete_Hime · June 7, 2016, 4:47pm

Just the big blue button [Download Autohotkey] I guess. If you have a 64-bit system you could consider downloading the 64-bit exe.

The script as it is now assumes your system (Notepad) and Chrome and Memrise are in English. Let me know if that’s not the case, perhaps after you’ve tested it and it doesn’t switch to Notepad and then back to Chrome again.

guestgulkan · June 12, 2016, 3:54am

Can someone save this important thread from the Indonesian forum? It details a script to generate google translate audio MP3s and then upload them to your course.

http://www.memrise.com/course/32267/2000-indonesian-words/thread/1347089/

Arete_Hime · June 13, 2016, 12:20pm

@guestgulkan see: [wiki] All the Userscripts

DW7 · June 14, 2016, 1:36pm

@guestgulkan < Have you tried a simple highlight, copy & paste to a word file? That seemed the quickest way for me when I saved pages and pages of Italian Forum posts.

cos · December 9, 2021, 3:07pm

Indeed, you are right, it just took them a long time to get the images so when I first looked at discussions I had gotten it to archive, I saw the images missing. But eventually, it got the images too. Here, for example, is the one I gave above as an example: Forums > General discussion: New Feature : Tapping Test - Memrise - with all the images.