TLDR: algorithm changed recently? please state changes as they very much affect learning retention. thank you.
Hello, i’m curious if any employees could comment on recent website algorithm changes. When I sampled the app previously, I noticed a large discrepancy with what I could remember versus the website. I felt at the time that, besides decreased “punishment” for missed answers, there was also a changed algorithm that had a gap at some key interval.
Over the last few months, there have been ambiguously addressed changes to the website algorithm. Right now, i’m experiencing a large drop in my memory when the review schedule gets to a particular distance from time initially learned. I know this sounds like i’m blaming memrise for my missed words, but i’m only asking if any staff could address the algorithm changes.
If possible, could changes be stated and if the website has been converted to the app algorithm, could you comment on what the old algorithm was? If I build some of my courses on a utility/app that allows me to tweak the algorithm, I would like to make something close to how the website was before.
This could of course be all in my head, but after years of using memrise, I’m quite in tune when a particular lesson is not as ingrained as others.
It would be very helpful if you could address this sooner rather than later. The reason for this is, if more words are planted now without properly supporting them, then a continued snowball of poor retention might happen. Words still stick with the current algorithm, but for myself personally, the old algorithm was practically flawless and is what in the beginning got me so addicted to the website: if the words were reviewed according to the timetable, they stuck like magic.
As a suggestion on the same topic, I think many users here would be overjoyed to assist in tweaking the algorithm if that’s what is being done. You could set up a different server to do so and testers could join.
For some more anecdotal evidence - I have found a similar thing happening. Some of it I’ve deduced is algorithm specific, and some is due to the algorithm impacting my memory on certain types of information that for me are hard to retain (like complex definitions of idioms). One day out of 30 I’ll suddenly be hit with 10-20 words I completely forget. Which is very strange. I feel like most apps are becoming more and more adaptive, like tweaking the algorithm based upon how they know we learn based upon some metadata of the things we’re learning, population statistics of retention rate of similar items, etc. So I feel like it’s constantly regressing, whereas this is quite a hot area so I don’t know why they aren’t pursuing it.
I also wanted to ask whether or not the “difficult words” functionality is a little bit strange. For me - the number practically stays exactly the same , and I’m unsure why. I am not a pro user , but wanted to have some sort of benchmark to know how many words I “know”… nonetheless for the past 6 months on one of my courses that number has stayed the same. It also seems like certain words I never seem to review.
I have the feeling that after the current push towards the younger-themed marketing started and the simplification to attract more users in the app, some of the “core value” that has remained in the company is the secret of the algorithm, hence the very light replies to anything touching on this. It’s kind of a shame because, first of all, algorithm adaptation is harder to pin down when a large majority of users are spamming reviews and doing speed and tap-tests wildly which doesn’t test retention in the same way. Especially with the hints added in the app. Mixing data from that group with website learners is going to give poor results.
Years ago when I noticed that speed review misses and correct answers actually affected the user’s algorithm (instead of just being a random quiz game), I was really confused as how they could mix something that’s kind of emphasizing guessing with, what I came to see as, the “cherished algorithm history of the user”.
Second, a lot of the learners on the site actually have an interest in the concept of memory and would be willing to dedicate time to helping with algorithm adjustments.
In the area of the difficult words, I’ve always felt a little sketchy in that department. Repeated words, and sometimes needing to enter multiple sessions for words to clear them. I still don’t think they’ve gone too far gone, since a lot of us haven’t left, but the amount of untapped potential is a little disappointing sometimes.
My hypothesis is 1) words marked as totally wrong (red) now are given back hours later than they did in the past and 2) without the all-typing script, getting a word wrong and only seeing it ONCE again in the session, and this time as a multiple-choice question, reduces its stickiness in your memory.
@TheFour-GatedDanzig yeah as far as short term memory is concerned, I could name a lot of areas where I don’t understand why the initial learning session isn’t imprinted a bit more solid. Something I remember memrise doing years ago. And with incorrect answers, I think you’re right in how it’s evolved. I specifically remember that when first trying to app when it came out, how much easier it was. I had to mentally prepare myself for a website learning session whereas an app session whizzed by like a bee fart.
I’m mostly able to alleviate the initial learning problems at least by making things harder on myself. I learn Chinese, so when i’m prompted with the English, i’ll close my eyes and visualize the character as best I can. When prompted with the character, instead of staring at the character, i’ll very quickly glance at it and then try to remember. I have this theory that when you quickly scan something, my brain takes a mental snapshot that I can break down. I used to work in a casino sitting on this card game called pai gow. My job depended on being able to read a 7 card spread, arrange them in my head by house rules, and compare them to a player’s 7 cards in a span of 1-2 seconds to prevent shady dealers from scamming. If I want to get a snapshot of a room in my brain, if I quickly flash my eyes at a wall or area and look away I’ll have a mental image of that area.
The area that really bugs me on memrise, is like what @roflcopterlol mentioned, in that there will be these times when there’s a sudden clear drop in retention. If memrise would respond to this, I think we could make the algorithm a lot stronger. I’d really like to look at the specifics of the current and older website algorithm from two years ago and see if there’s a changed/deleted review session somewhere at the 20-30 day mark. I get this feeling that the whole function of the website was Ben Whately’s baby, so to speak, and now that a marketing, demographic and “quantity” focus rules, I don’t think the algorithm is under the care of people who appreciate it fully. Someone like Ben would see an error and immediately be disturbed whereas newer employees are being briefed on the function of the website pertaining to it’s role in the app stores.
Alternately, the alrogithm problem is known but purposely left on the back-burner because it affects a lesser amount of people. That would be a lot sadder.