main
side
curve
  1. Welcome to the new boards! Details here!

Post technical questions and issues here.

Discussion in 'Communications' started by Grimby , Sep 2, 2012.

  1. LAJ_FETT

    LAJ_FETT Tech Admin and Collecting/Lucasfilm Ltd Mod star 9 Staff Member Administrator

    Registered:
    May 25, 2002
    What are you searching on?
     
  2. Sarge

    Sarge Chosen One star 6

    Registered:
    Oct 4, 1998


    So which user did you ban?


    Things were a little rough about 3 hours ago, a couple of bad gateway messages, but it seems smooth and fast right now.
     
    Master_Rebado likes this.
  3. LAJ_FETT

    LAJ_FETT Tech Admin and Collecting/Lucasfilm Ltd Mod star 9 Staff Member Administrator

    Registered:
    May 25, 2002
    Yeah, there was a server issue which was resolved. Things have been better since then.
     
    Master_Rebado and Sarge like this.
  4. Grahame

    Grahame Server Admin star 1 Staff Member Administrator

    Registered:
    Sep 3, 2012

    We're still talking to the support people at Vultr to find out A) exactly what they fixed and B) how long it had been a problem.
     
    Master_Rebado and Sarge like this.
  5. Grahame

    Grahame Server Admin star 1 Staff Member Administrator

    Registered:
    Sep 3, 2012
    ES is up, might be missing some more recent results since the indexes haven't been updated in a while.
     
  6. LAJ_FETT

    LAJ_FETT Tech Admin and Collecting/Lucasfilm Ltd Mod star 9 Staff Member Administrator

    Registered:
    May 25, 2002
    I noticed that it was up - fewer errors in the server log. (Just the 'shard failure' message that we normally get).
     
  7. V-2

    V-2 Jedi Master star 5

    Registered:
    Dec 10, 2012
    I don't want to jinx anything, but this forum seems to be working exceptionally well today!

    ETA: like this message just ...worked! Straight away!

    And I can read pages of threads! Even reply to posts within them! This is awesome guys, well done. :D

    ETA: And 'watched threads' loads in under 2 seconds!

    *tries 'new threads'*

    Practically instant!

    More of this sort of thing! :D
     
  8. JediMaster_Jen

    JediMaster_Jen Jedi Grand Master star 4

    Registered:
    Jun 3, 2002
    Ah, heaven! Working like a gem. =D=^:)^
     
  9. jcgoble3

    jcgoble3 Jedi Grand Master star 5

    Registered:
    Nov 7, 2010
    So I've noticed something new. For as long as I can remember on these boards, right now (a few minutes after midnight PT) the boards would be down every single night. I'm not referring to the general slowness and errors we've had at all hours for the past few weeks, but only to the specific nightly downtime right at midnight PT that has been going on for years (and has been frequently reported in this thread by people unaware that it's a normal, expected thing). This has been explained in the past as the nightly backup process needing to lock the database or something like that. Now tonight, as well as last night, I'm able to access the boards just fine. Did something change with the backups to where it's no longer necessary to shut down the boards while it runs?
     
  10. Grahame

    Grahame Server Admin star 1 Staff Member Administrator

    Registered:
    Sep 3, 2012

    So, here's the deal. On the RS forums we use a backup method called "MySQLHotCopy" because RS uses a MySQL setup where each database table is in a separate file and those files have to be locked in order to be copied. This takes down the system while the tables are locked. On the JC boards we have to use a different method (Percona Xtrabackup) because the database is in InnoDB format (all tables are in one HUGE file). Xtrabackup does NOT require that all tables be locked during the copy, so the boards should in theory stay up. However, when we first moved to Xenforo and set this up I'd never used InnoDB before and noticed that after a backup a bunch of files were missing (there are still some indexes and other things stored in single files outside of the huge InnoDB file). Therefore in order to make sure we got a good backup I had it set up to do a MySQLHotCopy as well as the Xtrabackup just to be safe.

    Now, during all of the recent unpleasantness I took a full copy of the Boards database for myself so that I could work on troubleshooting the slowness on my own PC without impacting you guys any more than you already were. In the course of my work I discovered that Xtrabackup does indeed back up everything it needs to do a full restore and I simply did not know how to "prepare" the backup correctly to be used. I tested this a bunch of times and sure enough, I could take a backup with Xtrabackup, prepare it for use and turn it back into a working database copy again. This means the MySQLHotCopy on the JC boards is no longer needed, ergo no table locking and no downtime.

    TLDR;I learned something new, and the backups now work without locking.

    The End.
     
    Sith-I-5, EHT, V-2 and 9 others like this.
  11. Ewok Poet

    Ewok Poet Force Ghost star 5

    Registered:
    Jul 31, 2014

    I guess this was a question for me? Sorry that I didn't see it last night.

    I'm searching story names, but AO3 hijackes the search result page, even when I type "jedi council" or similar. At this point, I wonder if Google is acting funny in this particular area of the world, since I do some SEO as a part of my job and I have never encountered anything like that before.
     
  12. LAJ_FETT

    LAJ_FETT Tech Admin and Collecting/Lucasfilm Ltd Mod star 9 Staff Member Administrator

    Registered:
    May 25, 2002
    When you search, follow your search terms with 'site:boards.theforce.net' (no quotes, no space before or after the colon). That should restrict your search to these boards.
     
  13. jcgoble3

    jcgoble3 Jedi Grand Master star 5

    Registered:
    Nov 7, 2010
    Cool! Thanks for figuring it out, and for all you've done over the past couple of weeks! :)
     
  14. Jester J Binks

    Jester J Binks Jedi Master star 4

    Registered:
    Dec 19, 2016
    Somebody noticed a problem there. I'm just reporting it.
     
  15. TiniTinyTony

    TiniTinyTony Jedi Grand Master star 6

    Registered:
    Mar 9, 2003
    When I click on Lit prefix to filter the threads in the Literature forum, the result is only 4 threads. A quick glance of page 1 shows that there are definitely more than four threads with that prefix. Thanks!
     
  16. LAJ_FETT

    LAJ_FETT Tech Admin and Collecting/Lucasfilm Ltd Mod star 9 Staff Member Administrator

    Registered:
    May 25, 2002
    Yeah, we've been discussing that over in the Forum Upgrade thread - there's a problem with prefixes at the moment.
     
    Ewok Poet and TiniTinyTony like this.
  17. TiniTinyTony

    TiniTinyTony Jedi Grand Master star 6

    Registered:
    Mar 9, 2003
    Yep, I just noticed that. Glad you guys are on top of it!
     
    LAJ_FETT likes this.
  18. gezvader28

    gezvader28 Jedi Grand Master star 5

    Registered:
    Mar 22, 2003
    how do I access the Senate forum ? every time I click on the icon it just says "no new threads to display"
     
  19. LAJ_FETT

    LAJ_FETT Tech Admin and Collecting/Lucasfilm Ltd Mod star 9 Staff Member Administrator

    Registered:
    May 25, 2002
    Yeah - tags are a problem right now. I know the US Politics thread has stayed pretty much on page 1 of Community. If you're after different threads than that one you'll have to scroll down through the other threads to find the one you want. You could try the Search Forum function up top - I haven't played with it yet so if it helps let us know here.
     
    gezvader28 likes this.