Category: Systems Administration

Short post: Most important command I always forget

December 30th, 2009 · The Fraser · Linux Administration, Systems Administration No comments - Tags: ,

Often times in the heat of a critical server restart the system decides to FSCK a very large filesystem. If I let it take its natural course it could be a day before its back (for 12TB file systems using ext3). So to stop it from fscking follow this procedure.

1) Restart system
2) When it begins FSCKing, it will start running a percentage counter. Crtl-C from this. It will dump you to Repair Filesystem mode
4) Once in repair after entering the root password you will need to edit fstab to comment out the partition. This fails as it is unable to edit the file as its Read-Only.
3) Run the command “mount -o remount /” or to be absolutely sure “mount -n -o remount,rw /”
4) Comment out the partition in /etc/fstab and restart the box.
5) Once the box has recovered uncomment the partition and mount it.
6) Schedule downtime for the box for fscking.

This doesn’t take into consideration the negative affects of running an ext3 filesystem without fsck for that long a period and cheating to get around it. But there are times when things just have to be fixed ASAP, this comes in handy.

RANCID and MultiContext Firewalls

September 1st, 2009 · The Fraser · Network Administration 3 comments - Tags:

I have deployed RANCID, a popular open-source network device config backup utility, in our environment. It works great especially when configured to svnsync to a master repository. The recent trouble I have found is RANCID doesnt support multi-context firewalls. There is no way to configure RANCID to detect multi-context mode, go to each context and download the config. Changing the Firewall so that each context has an admin address is seriously not going to work. So my only resolution was to create a script.

Continue reading »

Cisco Console Server

May 24th, 2009 · The Fraser · Network Administration No comments - Tags: , , ,

In an effort to provide a more secure way to access our network equipment in the event of an outage we setup a console server. We didnt purchase anything from a vendor but built a Cisco Console server. There is little documentation for creating this online so I am going to go through the steps of creating one.

Equipment used: below is the list of equipment needed to get started with the project

  1. Cisco 2621xm router
  2. NM-32a Async network module
  3. cab-octal-sync Async octal set of cables (68pin connector terminated with 8 RJ45 async console ports)
  4. cat5 cables of appropriate length and color (In my environment, color signifies function. This may not apply in your setup)

Connectivity Setup:

To start we are going to setup the devices, power it and assume the switches have been setup appropriately.

Continue reading »

Busy Life, New Project

April 6th, 2009 · The Fraser · Linux Administration, Network Administration One comment

I haven’t posted in awhile as life is pretty busy. I am almost done with the DataCenter Migration and we are greatly busy with looking for a Condo in the Bay Area (Perfect time to Buy). We found one condo that we are really interested in but it already has a few offers on it. So we can just hope and pray that they all fall through. My Wife got a new job recently too. Lawyers Assistant for a law firm, working her way to a Paralegal.

As for my new project, I am going to start working on a IT Assets Database. Mostly for the server side. This will be used to inventory all servers, predict average power consumption per server and site and hopefully find a good way to integrate it into a PXE/Kickstart setup. I have a few ideas on  the most efficient way to do this. I also partially want to the develop this as a Rails/Ruby app. I may end up developing a PHP app first as the guys here are so afraid of Ruby & Rails.

Updates will be more frequent in the future as work is slowing down a bit and the project is coming to a close.

Quick Post – DataCenter Pics

February 25th, 2009 · The Fraser · Personal, Systems Administration No comments

We have been working on building/migrating to a new DC for the past few months and this week the work has really started. I dont have pics of the latest systems I have racked, but the below photos are from earlier this month. I will be posting more photos as the building goes on. Currently I am only able to use the camera on my phone, so you are stuck with so-so quality.

Enjoy,

[picasa width="400" height="400" autoplay="0" showcaption="1"]http://picasaweb.google.com/aaronmfraser/DataCenterPics[/picasa]

Change Java Heap Size on a Mac

February 13th, 2009 · The Fraser · Systems Administration No comments

I do a lot of work with the Cisco ASA (Adaptive Security Appliance). The GUI interface relies on Java for any non-windows hosts. If you by chance have an ASA with an IPS module installed you will receive a Java Heap Size is too small error. To fix this error I have found no way on the mac to do this globally. So I edited the info.plist for the application that I was working on.

If you pop open the plist into an editor and make the following changes:


<key>jnlpx.heapsize</key>
<string>67108864,536870912</string>
.....
<key>VMOptions</key>
<array>
<string>-Xms67108864</string>
<string>-Xmx536870912 </string>

You shouldn’t run into the Heap Size issue again. #Note: this change only affects the app that you configured, this is not a global change.

Network Design

February 11th, 2009 · The Fraser · Network Administration No comments

Part of my tasks for the Data center Move is to rack, build, config the networking equipment. We have racked 99% of the network equipment and are now configuring everything. Part of the problem is the network hasn’t been accurately planned and the Firewall rules are for the most part incomplete. We have a lot of work ahead of us.

I keep saying I’ll get some pics up soon but I haven’t had the time to sit down and collect them all and post to flickr/Picasa (still haven’t picked which one I want to use).

Just an update

February 3rd, 2009 · The Fraser · Systems Administration No comments

The company I work for is building out a cage for our equipment and I am heading off much of the planning and foot work. Thus I am unable to post as often since I have little to no free time.

I will be posting images of the Datacenter changes soon.

MSExchangeIS #8217

December 17th, 2008 · The Fraser · Windows Administration No comments - Tags: , ,

Recently our Sr. Director of IT sent out a Calendar invite to the team for meetings to plan our major project. He sent this meeting to a few select developers and the Operations group. But when he sent to the Operations group, he didn’t expand the group and add each user. He added the Security Group. This caused a serious of errors in Exchange that didn’t permit any user on the Ops group to add the event to their calendars. After being asked to fix this when the person in charge of Exchange was less than adequate for the task, I went searching through the Event Viewer. Found many errors and reduced it to a few that stood out, Error #8217, #8260 & #3005.

#8217 pointed to the particular Calendar event and creator of the event. #8260 logged the inability to access this event using anything other than MS Outlook (OWA and Non-MAPI clients).

ESXi password changing Part #2

December 15th, 2008 · The Fraser · Vmware Administration No comments - Tags: , , ,

After finding all the recommendations online to not work. I was forced to completely rebuild the system. This means since its a Dell PowerEdge 2950 with PERC6/i, I had to delete all RAID Virtual Disks. The reason for this was a glitch I had found when first installing the box some 5 months ago. If you configure a ESXi server on a PERC Controller that has more than 1 Virtual Disk, it will not boot. You have to complete installation using one VDisk then successfully reboot it. Once its up, you have to reboot once more create any other onboard VDisks you need then boot completely. It will automatically detect the new drive and format it with VMFS for you.

All this to say I spent most of my day rebuilding an ESXi Host to recover from a lost password. Since this machine will be folded into our VirtualCenter in the near future, it might behoove me to just reinstall using ESX proper like the other servers. I will do some through testing of the host before had to see if there is anything to gain from using the lighter hypervisor over its far more robust RedHat bound ESX brother.