OSG site slow to respond?

[HNB]

What about a return of some official OSG t-shirts with profit going directly back into the site? That way we get something cool to show of the site and any money can go into helping with upkeep? Then it is optional and also gives users something cool to show their love? OSG hats? OSG buttons or pins?

I know that is pretty commercial, but you can be non-profit (free to use and free to enjoy) and still sell merch if the money goes into the site.

[GilmourD]

I made a change that might affect people receiving email notifications. Please let me know if you aren't getting emails you're expecting.

...it's still a PITA

Same, and it's so inconsistent.

A question for luau: Is this a private machine that the server is running on? A shared machine? A VM? Is the database and the forum instance running on the same box?

I ask because I'm starting to wonder if the hangup is network or server resource based.

It's a single VM with MySQL and Apache.

ATM, I think we just need more RAM but am still convincing myself of it.

How much RAM is allocated to the VM? I imagine you're running a Linux distro without a GUI, so it's pretty lean, but concurrent SQL queries will eat up RAM, I imagine. I mean... These days a regular desktop gets minimum 16 gigs in my mind. My home and work desktops both have 32 gigs. A Linux VM probably doesn't need that much, though.

When I became the SysAdmin at my current job (after the previous director was fired for doing nothing and lying to the superintendent that his pet project was completed when it wasn't even started) I discovered that one of our DOMAIN CONTROLLERS was configured as a VM with only FOUR GIGS of RAM. It was configured with AD, DHCP, and DNS, so it was getting hammered constantly.

[luau]

I made a change that might affect people receiving email notifications. Please let me know if you aren't getting emails you're expecting.

Same, and it's so inconsistent.

A question for luau: Is this a private machine that the server is running on? A shared machine? A VM? Is the database and the forum instance running on the same box?

I ask because I'm starting to wonder if the hangup is network or server resource based.

It's a single VM with MySQL and Apache.

ATM, I think we just need more RAM but am still convincing myself of it.

How much RAM is allocated to the VM? I imagine you're running a Linux distro without a GUI, so it's pretty lean, but concurrent SQL queries will eat up RAM, I imagine. I mean... These days a regular desktop gets minimum 16 gigs in my mind. My home and work desktops both have 32 gigs. A Linux VM probably doesn't need that much, though.

When I became the SysAdmin at my current job (after the previous director was fired for doing nothing and lying to the superintendent that his pet project was completed when it wasn't even started) I discovered that one of our DOMAIN CONTROLLERS was configured as a VM with only FOUR GIGS of RAM. It was configured with AD, DHCP, and DNS, so it was getting hammered constantly.

2GB. I realize it doesn't seem like a lot, but keep in mind it was perfectly adequate up until a week ago or whatever. Requirements for Windows machines with GUIs and Linux machines without are pretty different. I can't imagine trying to do anything with Windows with 4GB, not even Minesweeper.

That said, I'm sure we could throw hardware at it, virtualized or no, and make this go away. But are we really serving that much more content to that many more people all of a sudden? We really aren't.

[GilmourD]

But are we really serving that much more content to that many more people all of a sudden? We really aren't.

Well, that's why I was wondering what the setup was. Is there another VM on the same cluster that's sucking up resource time? Or is there maybe even a runaway process on the forum VM?

Honestly, without having my hands in the machine, I'm kinda just spitballing.

A "ping-t" doesn't even show anything but the occasional non-returned ping after like 10 minutes and the ping times are fairly consistent otherwise, so the server's ICMP is functioning correctly and consistently, but that does bypass DNS by going straight to IP.

Every time I see any delay in the waterfall it's in the SSL section of the resource load time, which is why I was initially thinking it's networking.

Interestingly, in trying to pull up a fresh waterfall I've refreshed the home page a whole bunch of times and haven't had any delays.

[luau]

But are we really serving that much more content to that many more people all of a sudden? We really aren't.

Well, that's why I was wondering what the setup was. Is there another VM on the same cluster that's sucking up resource time? Or is there maybe even a runaway process on the forum VM?

According to Digital Ocean the hypervisor is fine. There's no way I can verify that but there's no real reason not to believe them.

There is no runaway process. I'm so tired of looking at atop and its logs I could die. The only real problem I can turn up is some occasional excess page swapping with atop which Digital Ocean doesn't even cover in their guide. From the atop man page:

If the number of pages swapped out ('swout' in the PAG-line) is larger than 10 per second, the memory resource is considered 'critical'. A value of at least 1 per second is considered 'almost critical'.

So while RAM is an issue, it's hard to see how it's a big enough issue to cause what we're seeing. I'll take it up with Digital Ocean when I share my findings with them.

Whatever the case, the site is behaving very well ATM, so we've got that going for us, which is nice. And it's nice to chat about it with you, too, so thanks for taking the time.

edit:

[Embenny]

I've had a solid hour of full speed OSG and I'm so happy.

I realized I've been checking the site much less often with the loading issues, after one or two 30-second timeouts I often give up and wander away.

[luau]

I've had a solid hour of full speed OSG and I'm so happy.

I realized I've been checking the site much less often with the loading issues, after one or two 30-second timeouts I often give up and wander away.

I do the exact same thing, but with the added layer of not wanting my requests to make things worse for everyone else.

I apologize to all of you for this taking so long. It's supremely frustrating.

[GilmourD]

But are we really serving that much more content to that many more people all of a sudden? We really aren't.

Well, that's why I was wondering what the setup was. Is there another VM on the same cluster that's sucking up resource time? Or is there maybe even a runaway process on the forum VM?

According to Digital Ocean the hypervisor is fine. There's no way I can verify that but there's no real reason not to believe them.

There is no runaway process. I'm so tired of looking at atop and its logs I could die. The only real problem I can turn up is some occasional excess page swapping with atop which Digital Ocean doesn't even cover in their guide. From the atop man page:

If the number of pages swapped out ('swout' in the PAG-line) is larger than 10 per second, the memory resource is considered 'critical'. A value of at least 1 per second is considered 'almost critical'.

So while RAM is an issue, it's hard to see how it's a big enough issue to cause what we're seeing. I'll take it up with Digital Ocean when I share my findings with them.

Whatever the case, the site is behaving very well ATM, so we've got that going for us, which is nice. And it's nice to chat about it with you, too, so thanks for taking the time.

edit:

Yeah, it's like something snapped back into place and it's been working just fine.

And same! I'm so used to having to translate from tech-to-English in a school system that sometimes I forget how to talk straight tech. LOL

[GilmourD]

Oh, man. I think we jinxed it. Just posting my last reply had delays. D'oh!

[Embenny]

I do the exact same thing, but with the added layer of not wanting my requests to make things worse for everyone else.

I apologize to all of you for this taking so long. It's supremely frustrating.

I can't speak for everyone, but speaking for myself, you have nothing to apologize for. I appreciate all the voluntary effort you put in to keeping things running!

Shit happens. All you can do is try your best to deal with it.

[daemon]

I'm amazed this thing can even run as well as it has on 2GB, but if it's been that way forever then it seems like something else has changed.
* Is someone applying updates? (apt-get dist-upgrade or dnf/yum update)
* Is there any monitoring available that could show trends? (snmp, or running atop as a service)
* Are there any clues in the logs? (/var/log/messages, dmesg, etc)

[DeathJag]

I apologize

Nah man stuff happens! I thank YOU for all the work and energy you put into it. I wouldn't be surprised if it's some glitch in someone's (not you) "new" hardware that they are not disclosing. It's so random.

[luau]

I'm amazed this thing can even run as well as it has on 2GB, but if it's been that way forever then it seems like something else has changed.
* Is someone applying updates? (apt-get dist-upgrade or dnf/yum update)
* Is there any monitoring available that could show trends? (snmp, or running atop as a service)
* Are there any clues in the logs? (/var/log/messages, dmesg, etc)

There were no updates applied around the time the trouble started but the machine is being updated. I do have atop running as a service now.

There was no monitoring running at the time and I've found nothing of interest in the system logs, web server log, or mysql log.

[MayTheFuzzBeWithYou]

It appears to be veeeeery fast now.
I‘m happy that way. Please don‘t change it back!

[GilmourD]

It appears to be veeeeery fast now.
I‘m happy that way. Please don‘t change it back!