Elasticsearch Has Been Disabled

Discussion in 'Communications' started by Ramza, Dec 6, 2015.

  1. Ramza Administrator Emeritus

    VIP
    Member Since:
    Jul 13, 2008
    star 7
    As you may be aware, in the past couple of weeks we've been experiencing periodic and uncharacteristic amounts of slowdown. After discussion with Grahame and Philip, it's become apparent that at least some of this slowdown is due to the large amount of resources being dedicated to Elasticsearch, the background process that enables our handy on-site search function. Consequently, with an eye towards approving general board stability, we have disabled Elasticsearch, and are exploring alternative search methods that could be viable on the boards (E.g. integrated Google search, the default Xenforo search feature, etc.).

    In practical terms for you, the users, you should expect search to not function properly while we consider alternatives, as Elasticsearch uses a dedicated index that posts made following the move have been sorted under. While we realize this isn't an ideal status quo, we're hoping that there's an improved main board usage experience as the tradeoff, and we hope you understand that this came down to a difficult calculus between having accessible boards and having a functioning search, and thus we chose the former.

    While we're in this period of disabled Elasticsearch, please keep us abreast of errors that may be occurring as a result of that change in this thread. Also, our boards are fairly well-indexed by Google, so using their search engine combined with the tag "site:boards.theforce.net" should be a relatively useful substitute. For example, to search for uses of the phrase "Yarael Poof" on our boards, you can enter
    Code:
    Yarael Poof site:boards.theforce.net
    into Google's search bar. We will also strive to keep you posted on updates as we become aware of them, but keep in mind that we're coordinating this alongside the expected hit from the release of The Force Awakens, so we're not necessarily going to see fast developments on this.

    We apologize for the inconvenience this will obviously result in. If you have more questions, feel free to post them in this thread.
    Last edited by Ramza, Dec 6, 2015
    Sudooku, pronker, BigAl6ft6 and 3 others like this.
  2. Boba_Fett_2001 Force Ghost

    Member Since:
    Dec 11, 2000
    star 8
    So this affects post history right? Because if I look at mine I see one post I made today but then the next one is from 2012.
  3. Darkslayer Force Ghost

    Member Since:
    Mar 26, 2013
    star 6
    I noticed the slow down the other day, I was really confused seeing as we were still 2 weeks out from Episode VII. Thanks for the update @Ramza
  4. Ramza Administrator Emeritus

    VIP
    Member Since:
    Jul 13, 2008
    star 7
    Correct: your post history prior to the move was indexed during the move process, and thus still shows up as it's not part of Elasticsearch functionality, whereas posts made while Elasticsearch was enabled were indexed by Elasticsearch, and thus no longer show up. Getting those posts to show up in your history is something we're also investigating, but for now we're just looking to see if we get the stability improvements we want.
    Last edited by Ramza, Dec 7, 2015
    skypadme94 and Boba_Fett_2001 like this.
  5. GrandAdmiralJello Comms Admin + Moderator Communitatis Litterarumque

    Administrator
    Member Since:
    Nov 28, 2000
    star 10
    Cool. Are we still going down next week, or is possible that disabling indexing might help us stay above water? :p
    Last edited by GrandAdmiralJello, Dec 7, 2015
    jcgoble3 likes this.
  6. Ramza Administrator Emeritus

    VIP
    Member Since:
    Jul 13, 2008
    star 7
    Not sure, but it'd be a nice plus. :p We do actually have a working theory now about the crashes that was born out of observations made following turning Elasticsearch off, so who knows?
    jcgoble3 likes this.
  7. Admiral Volshe Force Ghost

    Member Since:
    Sep 2, 2012
    star 10
    Weird observation... mine still show up. In all tabs (postings, recent activity). Is that not supposed to happen? Do we have zombie elasticsearch? :p Or is it just something that *may* not work?
    Darkslayer likes this.
  8. Dandelo Force Ghost

    Member Since:
    Aug 25, 2014
    star 6
    shhh don't tell 'em Volshe :p
    Darkslayer and Admiral Volshe like this.
  9. jcgoble3 Jedi Grand Master

    Member Since:
    Nov 7, 2010
    star 5
    No, your recent posts, like everyone's, are cut off at the time Elasticsearch was shut off. Only your posts back to around the wee hours of yesterday morning show up; those posts have been indexed by the regular XenForo functions. Everything between the move and early yesterday morning, though, was indexed by Elasticsearch and only Elasticsearch, so it is currently missing. For those of us who have been around since before the move, like me, our pre-move posts do also show up because they were indexed by XenForo during the move process presumably before Elasticsearch was set up.
    Admiral Volshe likes this.
  10. Cryogenic Force Ghost

    Member Since:
    Jul 20, 2005
    star 5
    This is a real bummer.

    I know you guys are probably trying your best, but it feels like this board is leaking and gradually falling apart.

    The truncated posts have never been fixed (it'll be four years next March).

    And now, there's this.

    Oh, well. Maybe just another bump in the road, on the way to something more special.
  11. Ender Sai Chosen One

    Member Since:
    Feb 18, 2001
    star 10
    [IMG]

    THIS IS AN OUTRAGE!
    Darkslayer likes this.
  12. Sith-I-5 Jedi Grand Master

    Member Since:
    Aug 14, 2002
    star 6
    I didn't even know that we even had a functioning search! Doesn't seem that long since I was being told that Search wasn't working, so I had been using Google to look for old stuff on here.
  13. Admiral Volshe Force Ghost

    Member Since:
    Sep 2, 2012
    star 10

    That's what I was wondering - thanks. I didn't realise there was a separate index that the XenForo was capable of updating itself. :)
  14. PatttyB0123 RSA Latin America

    Manager
    Member Since:
    Apr 2, 2003
    star 6
    Thanks for the information about the issue.
    I did not have idea how that things work.
  15. jcgoble3 Jedi Grand Master

    Member Since:
    Nov 7, 2010
    star 5
    I notice that the number of online users is (currently, at least) back down to more normal levels similar to where it typically was prior to the TFA rush. If this persists, then with less load on the server, might it be possible to try turning Elasticsearch back on and see if the server can handle it now? Google is simply not an alternative for the types of searches I perform; I need, at the very least, the ability to filter by user, date, and/or thread and the ability to treat posts individually rather than just scanning the whole webpage, and I've been sorely missing these features.
    Last edited by jcgoble3, Jan 25, 2016
    Gamiel and Darth Downunder like this.
  16. jcgoble3 Jedi Grand Master

    Member Since:
    Nov 7, 2010
    star 5
    Hello? @Ramza? @LAJ_FETT? Anyone?
    Tim Battershell likes this.
  17. JoinTheSchwarz JC Head Admin & Community Manager

    Administrator
    Member Since:
    Nov 21, 2002
    star 8
    Chill out. :p There are some logistical factors involved that we are not ready to announce at the moment.
    Last edited by JoinTheSchwarz, Feb 5, 2016
    jcgoble3 likes this.
  18. jcgoble3 Jedi Grand Master

    Member Since:
    Nov 7, 2010
    star 5
    OK, thanks. As long as I know I'm not being totally ignored. :p :)
    JoinTheSchwarz likes this.
  19. Boba_Fett_2001 Force Ghost

    Member Since:
    Dec 11, 2000
    star 8
  20. JoinTheSchwarz JC Head Admin & Community Manager

    Administrator
    Member Since:
    Nov 21, 2002
    star 8
    Right now we are focused on updating the forum software, so any changes to the search system will have to happen after that.
  21. Chewies_bandolier Jedi Grand Master

    Member Since:
    May 5, 2002
    star 4
    just pointing out that with some form of function (even limited to the most popular boards), the quality of the discussions could be much better since we're currently just seeing the same things asked ad infinitum, in some cases and it is a major pain in the ass to go back through all the posts and find the ones manually, that have the answer..
    Scapro Tyler likes this.
  22. thejeditraitor Chosen One

    Member Since:
    Aug 19, 2003
    star 6
    any info on getting a new search engine or re-implementing the old one since traffic has gone down?
    Last edited by thejeditraitor, May 25, 2016
    TK327 likes this.
  23. Scapro Tyler Jedi Knight

    Member Since:
    Oct 17, 2015
    star 3
    I asked the same question in the technical question thread and was told it won't be until a forum upgrade is done.

    I don't know anything about the logistics/etc however I do see an awful lot of threads being deleted with the mods saying "Not appropriate a thread already exists"

    As a new comer that is really hard to do without a search function.
    jcgoble3 and thejeditraitor like this.
  24. thejeditraitor Chosen One

    Member Since:
    Aug 19, 2003
    star 6
    yeah it worked ok but wreaked havoc with the site apparently.
  25. anakinfansince1983 Nightsister of Four Realms

    Manager
    Member Since:
    Mar 4, 2011
    star 9
    I literally just locked a thread and redirected a user to an existing thread, and as such I'll say this: it's not a punishment, it's not meant to be embarrassing, and we get that you can't search for an already-existing thread.

    The modus operandi in locking duplicate threads is a little different now than it was when a quick use of the search function would turn up a thread on the topic--unless, of course, a user is trying to start a thread on "Adam Driver's hair" when there is a thread called "What do you think of Adam Driver's hair?" five threads below it on the first page.

    If you have done as much as reasonably expected (i.e. at least look on the first page or two) and don't find your existing topic, it's possible that we just know of an existing one from memory.