kensuf

HVAC update...

A month ago, I worked with the campus PPD A/C super to schedule a day when we could shut everything in the machine room down so that one of his crews could replaced the chilled water pump. That date was set for today, Tuesday, March 8th, 2011, beginning at 8AM.

Last week I got an email confirmation on the date / time for the schedule from PPD. There was a slight error on the start time (they had 7), so I emailed back with a "are you sure?". Their person wrote back saying that indeed the work would begin at 8, but they were emailing us announcing the window beginning at 7.

No problem. I wake up this morning at 5:30, come to work and begin shutting down 6 racks worth of server equipment. That includes 10 external disk arrays (total formatted usable capacity of around 80tb, raw in the 160tb range).

8:00 rolls around, no sign of PPD. I figure these guys are basically state workers, so I'm not overly surprised..

8:45, still no sign of PPD. I go grab a coffee.

9:15, still no sign of PPD. At that point I call over there and ask where the work crew is that's supposed to be changing the chilled water pump.

After a few minutes, I get a call back, "the work crew came over there and did the work yesterday."

"WHAT!?!?! You mean you guys shut down my chilled water and A/C on the 7th without notifying us? We have close to $500k worth of temperature sensitive equipment in there, and millions of dollars worth of research data! We spent weeks working out the outage and you guys did it a day in advance?"

"Can you hold on? Let me double check on that. Can we call you back?"

Five minutes later, some bubba calls me back. "Well sir, it was such a nice day yesterday, we figured we could go ahead and knock that project out and it wouldn't have been too much trouble."

HOLY CRAP.

We have thermal alarms on the equipment, and nothing tripped, but what if it had? Would these guys have even begun to know what to do if the audible alarms started going off? Would they have known who to contact? Can we say no?

Needless to say, we're scheduling a meeting with the A/C super, a director of PPD, myself, the chair of my department, and possibly the dean of my college that oversees infrastructure..

(so to tie this into the failure / training threads, I've been basically a mental zombie since around 10:30 this morning)

waterrockets 

Yep. That's the way it felt when I was developing PC games. You had to have a "minimum system" spec, but still push all the latest hardware if a customer had it. Scalable games blow.

When I switched to console (XBox, PS2, GameCube, etc) life got a lot easier.

FWIW: most Android app ideas don't require anything beyond Android 1.5 or so. Some of the higher-tech stuff does, but those apps don't make any money anyway (too long to develop stuff like AR when you look at how few people give a crap). That said, I've written AR that runs on 1.5...

Holy crap.

mattm

+0x10000000!

Major ouch.

ridethecliche

Damn dude!

Your worries are much greater than mine, but I had a headache today as well.

The program for the online conference that I run had a hiccup today and it left 10+ sites from all over the world staring at a blank screen for half an hour while I struggled to figure out what was going on. Unbeknown to me, the server has a mirror which is what I accessed that day and it created two mirror conferences with the same ID's which were impossible to tell apart.

I think they're making an excuse on their end and my boss is super frustrated with the system (as am I because I get blamed for it, even though this was totally out of my control). The words "1/3 of your salary comes out of a grant that funds these conferences, and if we give these responsibilities to xyz, then you're not going to work for 2/3 the pay" left his mouth.

Needless to say, I'm looking around at other options. Just.In.Case.

jwible

I heard a thing on NPR, I think it was, the other day about airlines getting rid of paper approach maps and having pilots use iPads instead. This frightens me if the pilots are also not required to keep paper charts on board. In private pilot check rides one of the first things the inspector does is remove the batteries from any flight calculators you may have.

ridethecliche

The computer that I got is a little excessive for what I need. But the price was right and the specs are grand.

Think, Upper mid tier gaming computer with a 23 inch HD monitor. My parents gave me a budget, so I used it to the max and am paying for part of the stuff myself. Think hexa core amd with a 512 mb or 1gb graphics card that's about mid range in terms of specs. I can't wait to get a good FPS for the computer. Though the first thing I'm going to do is install HL-2 and get CS-Source. It'll be excessively wonderful

Then my mom tried to give me money when I left home. I love her dearly but I refused. I told her that they were buying me a computer so I had more than my share of what was needed. I think the drugs got to her.

Wr, I understand what you mean for consoles. I bet it's MUCH easier there which is why many are claiming the 'death' of the PC as a game system. Sure the high end is higher, but it's annoying for developers and game consoles are awesome now-a-days. That's why I bought one, so I didn't have to worry about having a constantly up to date PC. The opportunity presented itself though, and so now I have a computer that should be game worthy for 2ish years and be fine for 'regular computer' use for atleast 4-5. I'm content.

Now if only the PS3 played PS2 games
I might have to steal my PS2 back from the sister once I'm done studying. There are too many good RPG's on that console (and the PS1) that I would love to play.

ridethecliche

They justified this by saying that the iPad can be reset in a matter of a minute max. If they have another one on board, then that gets rid of the problem all together. If you're worried about water, etc. Maps would be about as useful as an IPAD.

I think it's efficient, not to mention neat. They'll probably keep the maps onboard planes as a backup anyway. The planes are run by computers anyway, so if you're worried about that then...

waterrockets 

I still don't buy it as the only source. There are too many opportunities for bugs in such an application. They'd be fine most of the time, but it's the corner cases that concern me.

Regarding maps and water, there are fiberglass maps that won't tear can be read under water. Not sure what aeronautical charts are made from though.

mattm

Corner cases can definitely be interesting.

There was one fine example of a new fighter jet, that when it crossed the equator the plane would flip upside down every time.... now that's a bad corner case! Though the equivalent here would be the map flipping upside down, but still, the idea scares me.

waterrockets 

Exactly.

Enthalpic

Oh no a computer almost got too warm! In my experience complaints about potential damage to scientific equipment are going to fall on deaf ears. If you want something to be done it has to involve public relations or safety issues. “Holy crap” events involve HazMat teams and building evacuations….

In this old building I’m happy so long as water isn’t leaking directly onto our equipment and even that is often too much to ask.

Lastly could all that data really have been lost? If so, you guys have bigger problems.

shovelhd

kensuf, I'd say unbelieveable, but I've been in this business too long. At least the reserve capacity handled the temp rise. If we lose a chiller it goes to 95 degrees within 20 minutes and an alarm calls my cell.

kensuf

I guess you have no idea the amount of time and manpower that could be lost while we recovered everything.

Enthalpic

I'm not saying your problem wasn't real or serious, just that it won't be a big deal to others.

We had a HVAC issue (fume hoods) that disabled ~30 employees for 7 months straight. I thought heads would roll over that, they didn't.

shovelhd

I don't know the details of your operation, but I have been there during the aftermath of 9/11. It sucked on so many levels.

waterrockets 

Agreed that most people don't understand the costs involved in recovery. If it happens once or twice, it's quite an education.

bike_boy

Ah, RTO & RPO...

waterrockets 

Today I think I nailed my Android offline maps solution to scale imagery for missing zoom levels. Just need to test it tomorrow. The Mercator projection quad square tile system is really sweet. Math is pretty straightforward, and so is tile storage.

Just like the augmented reality stuff, management is going to need to pull me off these tasks, because I'm about to go all crazy with math-happy feature creep.

kensuf

Oof. Yeah I can't imagine: as a bystander more than 1000 miles away, that sucked on just an emotional level. Rolling up shirt sleeves and getting to work on bringing things back must have been tough on you guys.

shovelhd

Fortunately, we had 321 people in the North Tower, and 321 people got out. By 11:30am we were going 24x7 setting up temporary work spaces, networks, and rolling DR recovery for their locally stored data that was obviously destroyed. Unfortunately, it was my 2nd day on the job. I'm not afraid to say that I was scared sh*tless. We got it done, though, and they were up and running in 48 hours in an abandoned shopping mall. This is why you spend millions on backup, recovery, replication, and DR plans and drills, and why that effort can never really be appreciated until you need it.

waterrockets 

♪ ♫ I found the ♪ ♫ memory leak ♪ ♫
♪ ♫ I found the ♪ ♫ memory leak ♪ ♫

mattm

Hopefully you didn't have to find it via code review.. they make apps for that!

So, how did you track it down?

waterrockets 

Eclipse MAT --> List GC Roots.

Turns out Google Maps Android API has several fields that hang onto the parent Activty. Some other forums had the common bits listed and pulled out with reflection (no public access). Also realized that I wasn't unsubscribing from a listener that kept a reference to the callback method. I thought it was going to be a lot worse, but it only took a day to get it all cleaned up.

I hadn't used MAT before, so that was a learning effort that went well. At the game studio, we had our own badass logger that maintained the same interface for PC, XBox, PS, and Nintendo versions of the game. You could even rewind the stack time. So if a grenade exploded in a room, and the game crashed, you could rewind to when the grenade was created and examine the heap at that time. Really cool.

MAT is a fine tool though. I like it.

I need to start taking a look at IntelliJ though...

waterrockets 

So, I also found a bug today with the Spidey Sense and no source code. Let's see who can feel it too (kind of an easy one, but somehow I was the only engineer out of 10 or so to pick it out in the three minutes after I heard about it).

Phone app hits a server app. We have a built pair running on the Internet, and a different built pair running on a private network. Slightly different builds on both sides, for both apps.

For some reason, today, right before an important demo, both the Internet and private network systems failed to take new reports made from the phone. They both worked flawlessly yesterday, and continuously for the past several weeks. These two systems don't touch each other in any way at all, and they have completely different usage history and data, but they failed at the same time.

So, what's changed that sprung the bug? (e.g. what's it related to?)

jwible

Phone radio turned off. Server IP changed. Server build reverted.