tag:blogger.com,1999:blog-82259115647561418662024-03-13T17:38:32.509-04:00Linux QuirksJasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.comBlogger30125tag:blogger.com,1999:blog-8225911564756141866.post-8771923957907940832012-05-09T14:11:00.000-04:002012-05-09T14:11:08.569-04:00Fixing the Postfix Envelope Address<p>
I found errors in my <code>/var/log/syslog</code> indicating undelivered email because my MTA didn't recognize the envelope addresses. The Postfix <a href="http://www.postfix.org/ADDRESS_REWRITING_README.html#generic">address rewriting readme</a> provided a simple solution: use <code>/etc/postfix/generic</code> to rewrite those "bad" addresses to something acceptable. I did that. But, then I realized that I would need to "build" <code>generic</code> into <code>generic.db</code> (and restart postfix) before I could be sure the rewrites would take effect. A search brought me to <a href="http://www.unixwiz.net/techtips/postfix-makefiles.html">a page on Postfix Makefiles</a>; it had more information than I needed, but it did provide the important command. Running <code>sudo postmap generic</code> from <code>/etc/postfix</code> rebuilt my <code>generic.db</code>.
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-13797608277241893242012-03-20T21:27:00.004-04:002012-08-07T07:26:13.888-04:00Partitioning a new 2 Terrabyte disk<p>
To make sure it will be a while before we have to revisit my wife's backup strategy, I bought a 2TB drive and installed it in my machine to serve as a Samba network drive for her backups. The physical install didn't present any nontrivial problems.
</p>
<p>
I ran into the first puzzle when I booted-up the machine for the first time. After selecting the default boot kernel via grub I got an error message indicating a missing partition table or some such. Ack! Did I somehow damage my primary drive while installing the new one? Then, I realized---the order used by grub might have changed. I edited the boot commands, changing "hd1" to "hd0". Viola! Back to normal.
</p>
<p>
After booting-up, I ran <code>fdisk /dev/sdb</code>. I created a single primary partition for the entire disk, but then got a strange warning message:
<blockquote>
Partition 1 does not start on physical sector boundary.
</blockquote>
<br>
A Google search indicated that I might need to start at 8 or 64 or some such. But, when I tried that, I got the same warning. Some more searching led me to an <a href="http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/#tools">IBM article on Linux on 4KB-sector disks</a>. After finding the <a href="http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/#tools">fdisk section</a>, I followed their advice to turn off DOS compatibility (<tt>'c'</tt>) and change display units (<tt>'u'</tt>); when I again tried creating the partition, the warning disappeared.
</p>
<p>
Note: If I had been more observant, I would have noticed that <tt>fdisk</tt> told me about the <tt>'c'</tt> and <tt>'u'</tt> options when it started-up. Doh!
</p>
<p>
Note: I needed the disk UUID in order to make a proper /etc/fstab entry. The easiest way I found to get this is to run<pre>
ls -la /dev/disk/by-uuid/
</pre>
Each UUID is linked to its corresponding device.
</p>
<p>
Note: I was slightly upset that only <tt>1.7T</tt> of the <tt>1.8T</tt> was available immediately after formatting. I learned that this was probably due to the 5% reserved for system use. I lowered this to 2% using the following command:<pre>
sudo tune2fs -m 2 /dev/sdb1
</pre>
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-37242663727721056102012-03-18T14:32:00.002-04:002012-03-26T10:48:22.981-04:00Setting up Samba for Windows Backup<p>Until recently, my wife has been a happy Mozy customer. She didn't mind paying for the 50 gig plan because she runs a website and blog for her business and needs a reliable backup solution that she doesn't have to worry about. About a month ago, she got a notification that Mozy hadn't performed a backup for 7 days because she exceeded her quota. This was the first time she had ever hit the 50 gig limit. She opted to pay for an extra 20 gig quota, thinking it would last quite a while. A month later, she got the notice again about being out of quota. Then, she realized, videos take up a lot of space! I helped her start looking at other backup options---paying for an additional 20 gigs of space every month did not seem like a good plan to either of us.<br />
</p><p>After looking into various options, I realized that the best one might be free---using backup software on her machine to save her data to a samba share on my Ubuntu box. I followed the <a href="https://help.ubuntu.com/11.04/serverguide/C/samba-fileserver.html">Ubuntu instructions for setting up a Samba file server</a>. Here are two notes that would have saved me a lot of time:<br />
<ul><li>Try using a workgroup name of "WORKGROUP" unless you have explicitly changed it on the Windows machine. I saw mentions of "HOMEGROUP" on my wife's Vista box and tried that first, but the real workgroup name was "WORKGROUP".</li>
<li>To access the shared drive, enter "\\machinename". If there is a "Run" option in your Start menu, enter it there. For Vista, I entered this into a Start menu "Search files and programs" search box.</li>
</ul></p><p>I then went searching for Windows backup software. I found and tried <a href="http://www.cobiansoft.com/">Cobian backup</a>. After much fiddling and reading, I learned that "backup" software is not what I was looking for---I wanted software to (efficiently) keep a current replica of my wifes files. Cobian could only do this by running a "full" backup each run. I wanted the functionality of "rsync" which apparently is known as "file synchronization". Next, I discovered <a href="http://www.goodsync.com/">GoodSync</a> which seemed to have the right functionality. One annoying part is that it required my wife's Windows account to have a password in order to set up an automatic sync. Easy enough. At this point, I had the desired automatic file synchronization functionality as long as I had the "guest ok = yes" setting for the samba share.<br />
</p><p>Next, I wanted to require authentication for the samba share. I followed the <a href="https://help.ubuntu.com/11.04/serverguide/C/samba-fileprint-security.html">"Security = user" instructions provided by Ubuntu</a>, but then found that I could authenticate with my Ubuntu account, but not the one I had set up for my wife. Next, I created an samba user/password for my wife with <tt>smbpasswd</tt>. I could access the share, but GoodSync was giving me strage errors every time a automatic backup tried to run. Finally, I realized my wife's account didn't have permission to write to the share. I learned of the "admin users" samba option and added my wife's Ubuntu account to the list of admin users for the share. Finally, the automatic file sync worked without issue.<br />
</p><p>P.S. At some point, I tried mapping the samba share to a Windows drive letter (e.g. "Z:"). But, that didn't help and only created more confusing error messages. In fact, GoodSync had no trouble using the "\\machinemame" notation to reference the samba share.<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com1tag:blogger.com,1999:blog-8225911564756141866.post-65541510186089058642011-07-26T08:54:00.002-04:002011-07-26T08:56:34.564-04:00How to Create a Web-Based Video Converter<p>
My wife <a href="http://helenrennie.com/kitchen/">teaches cooking classes near Boston</a> and <a href="http://www.beyondsalmon.com/">blogs about food</a>. Recently, she's become interested in video. She's already made a few videos using Windows Movie Maker (WMM), like this one on <a href="http://www.beyondsalmon.com/2010/06/how-to-dice-onion-video.html">how to dice an onion</a>. But, she's finding that free movie editing software just doesn't cut it. The kicker was when she published a video, learned that the volume was too low and couldn't find any way to raise the volume in WMM---it only allows limited volume adjustment.
</p>
<p>
She had already started researching video software options and settled on <a href="http://www.sonycreativesoftware.com/vegassoftware">Sony Vegas</a>. Yes, people complain that it crashes and/or runs slowly, but those same people are trying to process relatively large movies. My wife is looking to publish 5 minute non-HD videos on <a href="http://www.youtube.com/">YouTube</a>. So far, Sony Vegas has worked well for her. The one problem she's encountered is that Sony Vegas can't import Flip Video <a href="http://en.wikipedia.org/wiki/Audio_Video_Interleave">AVI</a> files. She found a free Windows-based converter. But, it leaves a blatant watermark and screwed-up a half-second of the audio track.
</p>
<p>
I figured this would be an easy problem to solve with Linux software. Sure enough, a bit of searching and I discovered <a href="http://www.ffmpeg.org/">ffmpeg</a>, which is available in <a href="http://packages.debian.org/stable/ffmpeg">Debian</a>. After a few minutes of man-page reading, I had a command-line to perform the conversion:
<pre>ffmpeg -i myvideo.avi -target ntsc-vcd myvideo.mpg</pre>
Note that this generates <a href="http://en.wikipedia.org/wiki/NTSC">NTSC</a> video. If you're in Europe, you might want PAL, which you'd get by changing the target to <tt>pal-vcd</tt>.
</p>
<p>
But, this wouldn't cut it. My wife didn't want to have to copy to Linux, convert, then copy back or wait for me to get home just so she could start editing a video. So, I set to work on creating a web-based converter. Creating a script to upload the video is straightforward and easy to find. Here's an example of <a href="http://www.tizag.com/phpT/fileupload.php">file upload HTML and PHP</a>. But, what this page doesn't tell you about are the hard limits PHP has on file and memory sizes. Here is a <a href="https://bugs.php.net/bug.php?id=13245">PHP "bug" which describes the max file size ("exceeds the limit of 8388608 bytes")</a> problem I quickly encountered. Sniper provides the config settings that need to be edited:
<pre>post_max_size = 256M
upload_max_filesize = 256M
memory_limit = 256M</pre>
I modified these settings in <tt>/etc/php5/apache2/php.ini</tt>, restarted my web server, and then was able to upload video files larger than 8 megs. The final question was how to push the converted video back to my wife's web browser. For some reason, all the pages I found on how to upload a file didn't mention anything about the possibility of pushing binary data back to the user. Finally, I stumbled upon the <a href="http://php.net/manual/en/function.readfile.php">PHP readfile function</a>. Occasionally, PHP is nice in that it provides tools and examples for what you probably want to do, like push an entire file to a user's web browser. The readfile manual page provides a full example for how to do this, including the necessary HTTP headers and proper output buffer management.
</p>
<p>
Here's what I ended-up with. Note that this script is unsafe due to the fact that it executes a shell command. Also, it relies on /tmp being the usual "temp" directory and can fail if a file already exists with the name $outfile. But, it serves it's purpose for me.
<pre><?php
if ($_FILES["video"]["name"]) {
$pattern = '/(.+)\.avi$/i';
$replacement = '${1}.mpg';
$outfile = preg_replace($pattern, $replacement, basename($_FILES["video"]["name"]));
$cmd = "ffmpeg -i " . $_FILES["video"]["tmp_name"] . " -target ntsc-vcd " . $outfile;
chdir("/tmp");
shell_exec($cmd);
header('Content-Type: application/octet-stream');
header('Content-Disposition: attachment; filename=' . $outfile);
header('Content-Transfer-Encoding: binary');
header('Expires: 0');
header('Cache-Control: must-revalidate, post-check=0, pre-check=0');
header('Pragma: public');
header('Content-Length: ' . filesize($outfile));
ob_clean();
flush();
readfile($outfile);
unlink($outfile);
exit;
} else {
?>
<html>
<body>
<form enctype="multipart/form-data" action="video.php" method="POST">
Choose a file to upload: <input name="video" type="file" /><br />
<input type="submit" value="Upload Video" />
</form>
</body>
<?php
}
?></pre>
</p>
Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com1tag:blogger.com,1999:blog-8225911564756141866.post-75349283650072125352011-07-19T11:31:00.001-04:002016-09-29T20:56:15.220-04:00How to Create a GMail Filter Based on Reply-To<p>
Today, I found myself trying to filter a message in <a href="http://www.gmail.com/">Gmail</a> based on the "Reply-to:" header. "Reply-to:" was the only user-friendly header that clearly distinguished it from other types of messages I receive. I knew that Gmail allows list-based filtering using a "list:" prefix in the "Has the words:" field. But, I haven't found a resource to tell you what, if any, other prefixes are allowed. I tried "reply-to:" without success. Then, I searched and found <a href="http://www.google.com/support/forum/p/gmail/thread?tid=5cb50e55319132ce&hl=en">another poor soul with the same dilema</a>. A bit more futzing and I discovered that "replyto:" was the correct prefix. Unfortunately, the Google help forum won't let me post a reply (!) So, I am hoping that romadatnvwisp will read this post and learn how easy it is to filter based on the "Reply-to:" header. Maybe he's figured it out himself?
</p>
<p>
To recap... lets say you want to filter messages with a Reply-to: address of "me@sample.com". Here's how you do it:
<ol>
<li>Click the gear icon in the upper-right-hand-corner of Gmail, then select "Mail settings"</li>
<li>Click the "filters" tab, scroll to the bottom and click "Create a new filter"</li>
<li>Type "replyto:me@sample.com" in the "Has the words" field</li>
<li>Click "Next Step" and finish creating the filter; note that matching messages should appear below after clicking "Next Step"</li>
</ol>
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com6tag:blogger.com,1999:blog-8225911564756141866.post-29164620889843797952011-07-10T17:08:00.003-04:002011-07-10T17:08:55.049-04:00How to Add Share Buttons in BloggerGoogle has redesigned the Blogger administration interface. As a result, the <a href="http://buzz.blogger.com/2011/06/add-1-button-to-your-blog.html">instructions posted by Blogger a month ago on how to add the "+1" button</a> are no longer valid. "Design > Page Elements" no longer exists. Instead, you need to click "Layout" then click the tiny "Edit" link at the bottom right-hand corner of the "Blog Posts" box. This will give you a list of post option. Make sure the "Show Share Buttons" checkbox is checked, then scroll to the bottom and hit "Save".<br />
<br />
Didn't work? That's expected if your blogger has existed for a while (more than 1 year?). Fortunately, this <a href="http://www.betatemplates.com/2010/06/official-blogger-share-buttons.html">Blogger Templates post on how to add share buttons describes how to hack it</a> (scroll down). This trick worked nicely for all but one of the blogs I administer. Unfortunately, for my wife's <a href="http://www.beyondsalmon.com/">Beyond Salmon</a> blog which was created six years ago, it only sort-of worked---the buttons came out very small and the "+1" button didn't appear (even after I moused over the other share buttons, which was the only way "+1" would appear on some other blogs). Bleh.<br />
<br />Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-26099721869019512602011-07-01T16:41:00.001-04:002011-07-04T09:34:04.490-04:00I Want My Old Google Calendar Back!<a href="http://www.beyondsalmon.com/">My wife</a> woke up this morning to discover that her Google Calendar had been transformed via Google's effort to <a href="http://googleblog.blogspot.com/2011/06/evolving-google-design-and-experience.html">evolve the Google design and experience</a>. I fail to see how this effort is a good idea. The result is that my wife can see <i>fewer</i> events on her "Month" calendar (which is the primary view she uses). She used to be able to see six events per day. Now she can see four and she has to click the "more" link to see all events on a day with 5 or 6 events. It seems that this is due to wasted space. There is more padding/spacing around <i>everything</i>, including day-of-the-week names and date numbers. What's the point of spreading-out everything in the display if it distances you from useful information without improving usability? I'm at a loss. Fortunately, it's easy to revert to the old UI---click the gear icon in the upper-right-hand-corner, then select "Use the classic look".<br />
<br />
Disclosure: I joined Google via the <a href="http://googleblog.blogspot.com/2011/04/ita-software-acquisition-cleared-for.html">ITA Software acquisition</a> and have no involvement in the above-discussed UI redesign.<br />
<br />
<b>Update 7/4/11</b>: Apparently, <a href="http://areallybadidea.com/google-calendar-redesign">I'm not the only one who thinks the Google Calendar redesign is a bad idea</a>.<br />
<br />Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com1tag:blogger.com,1999:blog-8225911564756141866.post-71792554627039550122011-06-07T07:30:00.004-04:002011-06-07T08:16:12.144-04:00Using Multiple Google Accounts in Chrome<p>Now that I'm an employee of <a href="http://www.google.com/">Google</a>, I have two Google accounts, corporate and personal. Google has made it relatively easy to manage two accounts with "account switching". If I'm in Gmail, I can easily switch between accounts by clicking on my email address in the upper-right-hand corner of the screen, then selecting "switch account" followed by the account I want. But, the choice of account isn't completely persistent. Occasionally, I find that when Gmail and other Google applications switch users without any input from me. This seems to be especially common when I return to the computer after an extended break (e.g. overnight). One of the google applications will ask me to re-authenticate and this will screw-up the choice of account in other tabs. Also, it occasionally switches accounts on me when I click a link.<br />
</p><p>One obvious solution is two use two different browsers, e.g. Chrome for corporate and Firefox for personal. But, I feel that this is worse than the occasional random account switch that I get by using a single Chrome instance due to differences in the UI. A better solution seems to be separate profiles. Chrome allows you to specify the location for your user data. So, you can run two instances of Chrome, one with the default location and one with a custom location:<br />
<pre>$ mkdir ~/.config/personal
$ google-chrome --user-data-dir=~/.config/personal
</pre>If you then use one account exclusively with each instance, there should never be any switching issues.<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-71628728584038373052011-04-05T06:38:00.000-04:002011-04-05T06:38:22.166-04:00Custom Search in Chrome<p>
At home, I have little use for domain-specific search engines---Google usually gives me the right answer. But, at work, Google doesn't have access to much of the information I need. Yes, we have an intranet search, but it is of little-to-no help when I'm looking for a particular bug, ticket or code revision. Firefox has "bookmarklets" which let you trigger a custom search based on a keyword. Though it's less obvious, Chrome lets you do the same thing via its "search engine" configuration. In the "Manage Search Engines" window, enter a keyword and a url and a "%s" for where the search string should be substituted. Then, finding a particular bug can be as easy as typing "b 12345" in the omnibox. I got this tip from <a href="http://lifehacker.com/#!5476033/how-to-set-keyword-bookmarks-in-google-chrome">Lifehacker</a>.
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-23545024225190134682011-04-01T10:30:00.001-04:002011-04-01T10:31:40.922-04:00Nohup or Disown?<p>
In the past when I've started long-running jobs, I've used <code>nohup</code>. But, what if you forget to use <code>nohup</code> or find that a job is taking longer than expected? Today, I learned about <code>disown</code> which allows you to easily prevent a process from being killed when the parent is killed. <a href="http://blog.ksplice.com/2011/03/disown-zombie-children-and-the-uninterruptible-sleep/">Ksplice provides a detailed tutorial on using <code>disown</code></a>. The quick-and-dirty version is:
<pre>
$ long-running-process.sh
^Z
[1]+ Stopped sh long-running-process.sh
$ bg
$ disown %1
</pre>
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-72641191565731144102011-03-25T10:42:00.000-04:002011-03-25T10:42:38.269-04:00XChat Configuration<p>
I'm an IRC newbie even though I've been using email for 15+ years. IRC seems great for those issues that benefit from the experience of many but aren't large or important enough to warrant hitting the mailboxes of 100 people. It also seems useful for determining the magnitude of an issue. Anyway, now that I'm using IRC, I wanted to make my client of choice, <a href="http://www.xchat.org/">XChat</a>, less annoying to use:
<ul>
<li>XChat highlights a channel when any new messages show up, including join/quit. So, as long as join/quit messages are displayed, the channel highlight wasn't very informative. I learned that <a href="http://xchat.org/faq/#q211">it's easy to disable join/quit messages</a>. Note that in in XChat 2.8.6, the "hide" option is under "Settings".</li>
<li>By default, XChat starts with no server connection and no channels. You can get it automatically connect and open channels via the "XChat"/"Network List..." menu. "Edit..." your server, check "Auto connect to this server at startup" and click the button next to "Favorite channels" to provide a list of auto-connect channels.</li>
</ul>
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com1tag:blogger.com,1999:blog-8225911564756141866.post-84010704066575030852011-03-22T21:55:00.001-04:002011-03-22T21:56:03.900-04:00Deleting Old Files<p>
Deleting old files should be an easy task, right? <code>find</code> can locate them and <code>rm</code> can delete them. But, what's the right combination? <code>xargs</code> must be useful as it converts newline-separated data into space separated data, but the first thing I tried didn't work because there were spaces in some of my file names:
<code><pre>
$ find -mtime +90 | xargs ls
ls: cannot access ./download: No such file or directory
...
</pre></code>
The <code>xargs</code> man page provides one solution:
<blockquote>
...filenames containing blanks and/or new‐lines are incorrectly processed by xargs. In these situations it is better to use the -0 option, which prevents such problems. When using his option you will need to ensure that the program which produces the input for xargs also uses a null character as a separator. If that program is GNU find for example, the -print0 option does this for you.
</blockquote>
Another option is to tell <code>xargs</code> to use newline as the delimiter. 'course, I'd recommend that you first try <i>listing</i> the files to see what you'll be deleting. Hence the <code>ls</code> in these examples. You should change to <code>rm</code> when you're ready to destroy.
<code><pre>
$ find -mtime +90 -print0 | xargs -0 ls
$ find -mtime +90 | xargs -d "\n" ls
</pre></code>
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-18884083499946974632011-03-10T11:25:00.003-05:002011-03-24T07:25:15.262-04:00Google Chrome 10: Flash Out of Date<p>
If you're like me and just upgraded to Google Chrome 10 (10.0.648.127), you may find that you get Flash is "Out of Date" error messages. IIUC, Chrome 9 used it's own build-in flash player, but Chrome 10 has switched back to using the Adobe one. Also IIUC, Chrome 10 expects Flash 10 and will complain if the available flash is 9 or older. At first, I thought I was out-of-luck since <a href="http://kb2.adobe.com/cps/839/cpsid_83950.html">this Adobe TechNote says that Chrome's flash is built-in</a>, but I found that after uninstalling (Adobe) flash, not even the "Run this time" option would work. So, I tried reinstalling Adobe flash. It grabbed version 10 of Adobe flash, and, Poof! Flash was working again without the warning message. This even though I'm on Debian oldstable (5.0 aka lenny). So, if you run into this issue, I'd recommend installing Adobe flash 10. It appears that on Debian-like distributions (e.g. Ubuntu), this is as easy as:
<pre>
$ sudo apt-get remove libflashplugin-nonfree
$ sudo apt-get install libflashplugin-nonfree
</pre>
</p>
<p>
<b>Update (3/24/11)</b>: Users on the <a href="http://www.google.com/support/forum/p/Chrome/thread?tid=4525b12dee005f88&hl=en">Google Help Forum</a> have suggested checking <code>chrome://plugins</code> to see if Chrome is using the Firefox's outdated plugin (<code>/home/username/.mozilla/plugins/</code>). If so, try deleting the outdated plugin and restarting Chrome.
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com1tag:blogger.com,1999:blog-8225911564756141866.post-88886188700851174922011-02-23T11:43:00.003-05:002011-02-23T11:44:36.991-05:00Reloading the Mutt Configuration File<p>
I find it a bit annoying that I have to re-enter my password twice every time I (re-)start mutt when I want to update my configuration file. Well, it appears I don't need to be doing restarting mutt. The <a href="http://wiki.mutt.org/?MuttGuide/Syntax">MuttWiki describes a way to re-load the configuration after mutt is running</a>. Type the following into mutt after you've changed your muttrc file:
<code><pre>:source /path/to/your/muttrc</pre></code>
Note that shell expansions may not work---you should spell out the full path.
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-11010165303307500862011-02-07T11:13:00.001-05:002011-02-07T11:26:34.447-05:00Too many open files<p>Certain applications, such as a web server with lots of database connections, require a large number of open files. Most Linux systems are, by default, configured to allow relatively small number of open files, e.g. 1024. How to change this limit isn't as obvious as one might hope. <br />
</p><p><code>ulimit</code> will show you current limits and let you change limits <i>for the current session</i>. But, one rarely cares about a temporary change. For a permanent change, one must realize that these limits are in place for security purposes---so that it is difficult for a single user to bring down the entire machine. So, the limits are configured in <a href="http://linux.die.net/man/5/limits.conf"><code>/etc/security/limits.conf</code></a>. Adding the following lines to <code>/etc/security/limits.conf</code> should help if you are having "too many open files" troubles:<br />
<pre>* soft nofile 16384
* hard nofile 65536
</pre>Note that this is also the place to "unlimit" the number of processes a user can run, e.g.:<br />
<pre>* soft nproc 4096
* hard nproc 16384
</pre>Note that a "soft" limit is the limit a user will get when starting a shell. The "hard" limit is the highest limit they can set without "root" privileges.<br />
</p><p>Note that when checking limits using <code>ulimit</code>, soft limits are shown by default. Use the <code>-H</code> option to get hard limits. The <code>-a</code> option shows "all" limits. So, run the following two commands to see soft, then hard limits, respectively:<br />
<pre>ulimit -a
ulimit -a -H
</pre></p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com1tag:blogger.com,1999:blog-8225911564756141866.post-21436824788920905212011-01-25T10:55:00.000-05:002011-01-25T10:55:19.909-05:00Indenting python code<p>Since python determines scoping by indentation, it's imperative to be able to indent and de-indent blocks of code. I just learned how to do this in emacs: <code>M-x python-shift-right</code> or <code>M-x python-shift-left</code>. Note that you have to be in python-mode for these to work. Here's a <a href="http://superuser.com/questions/173765/emacs-how-to-indent-unindent-region-of-python-code-by-4-spaces">discussion of all the possible ways to do it</a>. A good, non-python-specific solution is <code>C-x TAB</code> which calls <code>indent-rigidly</code>, but you need to give it an argument to indent more than a single space.<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-61916826107896292682011-01-16T17:40:00.002-05:002011-02-23T14:22:36.743-05:00Writing udev rules<p>One annoying thing about having a USB-connected <a href="http://www.davisnet.com/weather/products/vantagepro.asp">Vantage Pro2 weather station</a> is that, occasionally, there is sufficient noise in the line that Linux drops the connection and immediately reconnects. The immediate reconnection is good. What's annoying is that Linux usually assigns it a different device name upon reconnection. So, if I configure my weather station software, <a href="http://www.weewx.com/">weewx</a> with the <code>ttyUSB0</code> device name, I have to change the configured name after every reconnect (every few months) and risk losing data (though considering that the <a href="http://www.davisnet.com/weather/products/weather_product.asp?pnum=06510USB">VP2 data logger</a> keeps up to 2560 records, or 8+ days with a 5 minute interval, it's a small risk).<br />
</p><p>The solution to this problem is to use a custom <a href="http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html">udev</a> rule. Udev allows you to do a variety of things with the <code>/dev</code> directory, including providing a single, consistent name for each USB device you connect (no matter what <code>/dev</code> name it gets assigned by the kernel). The following rule creates a symbolic link from <code>/dev/vpro</code> to the current device name for my VP2:<br />
<pre>ACTION=="add", ATTRS{interface}=="CP2102 USB to UART Bridge Controller", SYMLINK+="vpro"</pre>This says to add the "vpro" link whenever a device with a matching "interface" attribute is "add"ed (i.e. connected/attached). What's a little tricky about adding a udev rule is figuring out what attribute to match against so that the rule triggers on the device you want (and only that device). For that, you need to know what device name the kernel assigns (what <code>/dev</code> entry), <br />
</p><p>To find out what to match against, you need to get the information about the device. The following command provides that. You will need to replace <code>/dev/ttyUSB0</code> with the device name that you are interested in.<br />
<pre>udevadm info --attribute-walk --path $(udevadm info --query=path --name=/dev/ttyUSB0) </pre>The command will yield a hierarchy of information about the device. Information at the beginning is probably too minimal to be useful for matching; information at the bottom is probably too general and will match too many different devices. Look for something <i>towards</i> the top which sounds like it might describe the device. Use the key and value to write a rule like the one I gave above. Put this rule in a file named like <code>66-vpro.rules</code> in<br />
<pre>/etc/udev/rules.d</pre>Note that the number must be two-digit and the extension must be <code>rules</code>. Make sure that the file is world-readable.<br />
</p>
<p>To test your rule, restart udev:
<pre>sudo /etc/init.d/udev restart</pre>
then unplug your device of interest, wait a second, and plug it back in. You should be able to see the effect of your rule. In my case, udev creates a <code>/dev/vpro</code> symlink to the kernel dev entry (e.g. <code>/dev/ttyUSB0</code> for my VP2).<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-39766968762864734932011-01-12T10:06:00.000-05:002011-01-12T10:06:56.052-05:00Parallel Programming<p>My first introduction to parallel programming was the summer of 1994 at the <a href="http://en.wikipedia.org/wiki/Pennsylvania_Governor's_School_for_the_Sciences">Pennsylvania Governor's School for the Sciences</a>. For my final project, we wrote Fortran77 code with an MPI library to parallelize neural network code for image recognition on a Cray T3D supercomputer with 256 processors. It was cute & fun, but neither terribly practical nor useful :-)<br />
</p><p>Four years ago, I had my second parallel programming experience, writing a recommendation engine for <a href="http://www.stylefeeder.com/">StyleFeeder</a>. This work was in Java utilizing threads to take advantage of multiple cores on a single machine. I got to see performance per core degrade significantly as the number of cores increased and the joy of the Java garbage collector (which would lock-up the entire program for minutes at a time). I've heard that Java GC performance continues to improve...<br />
</p><p>My most recent experience with parallel programming could be considered the most primitive one (single-threaded python code). But, considering the difficulty of parallel programming, I'm not convinced that it's any worse than the other frameworks. We use <a href="http://twistedmatrix.com">twisted</a> to simplify communication between processes and since twisted is an event loop (single thread), programming is greatly simplified---no need to worry about your code being interrupted anywhere.<br />
</p><p>A colleague of mine just discovered <a href="http://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html">Is Parallel Programming Hard, And, If So, What Can You Do About It?</a> by Paul E. McKenney. The table of contents piques my interest. I hope I'll find time to read it...<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-49167388533545736552010-12-23T13:51:00.005-05:002011-01-14T15:10:40.311-05:00Subversion: How to Commit a Directory Without Committing its Contents<p>I updated the <code>svn:externals</code> property of my project root directory to pull-in updated code from another project. But, I had local changes that I didn't want to commit. The vanilla <code>svn commit .</code> tried to commit all my changes, not just the property ones. The answer was simple: addition of the <a href="http://svnbook.red-bean.com/en/1.5/svn.ref.svn.html"><code>--non-recursive</code> (<code>-N</code>) option</a>. This option, however, is deprecated.<br />
</p><p>It appears that the <code>--depth</code> option is the replacement. The <a href="http://svnbook.red-bean.com/en/1.5/svn.advanced.sparsedirs.html">sparse directories</a> section of the SVM manual provides details about possible <code>--depth</code> arguments (which are lacking from the options manual page). Note that the <code>--depth</code> option would also be useful for trimming large sections of a checkout that are rarely/never used.<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-25316552330369180592010-12-11T16:30:00.003-05:002010-12-11T16:33:16.429-05:00Faster Brother Printing<p>The biggest annoyance I've had with my <a href="http://www.brother-usa.com/printer/modeldetail.aspx?PRODUCTID=hl2170W">Brother HL-2170W laser printer</a> is the speed at which data is sent and processed. Printing a one-page Google Maps directions would take minutes. Now, it takes about 15 seconds. What made the difference? Drivers, of course.<br />
</p><p>When I installed the printer, I selected an appropriate driver from the foomatic database when setting-up CUPS. What I didn't do was to check to see whether there were special Brother drivers which weren't installed by default. It turns out that there are quite a number of special Brother drivers which aren't installed by default. This <a href="http://ubuntuforums.org/showthread.php?t=848075">thread</a> directed me to open up <a href="http://packages.ubuntu.com/en/maverick/synaptic">synaptic</a> and search for Brother. Sure enough, there were at least two laser driver packages which I didn't have installed:<br />
<ul><li>brother-lpr-drivers-laser</li>
<li>brother-lpr-drivers-laser1</li>
</ul>After I installed them and the dependencies, I opened the Administration/Printing GUI and selected the "Change..." button for "Make and Model" of my Brother printer (select "Properties"). My printer was using the "Foomatic/pxlmono" driver. I changed this to the "Foomatic/ljet4" driver. Immediately, I found printing to be substantially faster. I retried printing a map that had taken minutes earlier in the day. It came out immediately after the printer had warmed up, no more endless "sending" status messages.<br />
</p><p>P.S. Note that if you don't have a laser Brother printer, you'll need to search through the driver package names to find one that matches your printer. Look for packages that start with "brother-lpr-drivers". The last word in the name gives you a hint of the type of printer it provides support for. In synaptic, you can click on a package name to see the description.<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-13179360839245725002010-11-23T08:20:00.001-05:002010-11-23T08:20:58.679-05:00libeatmydata<p>Apparently, <a href="http://en.wikipedia.org/wiki/Dpkg">dpkg</a> is so <a href="http://en.wikipedia.org/wiki/Fsck">fsck</a>-happy and <a href="http://en.wikipedia.org/wiki/Ext4">ext4</a> is so poorly optimized for <code>fsck</code> that <a href="https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/570805">a new Ubuntu install can take twice as long using ext4 as it would with ext3</a>. Also, apparently, the solution to this is a wonderful little library called <a href="http://www.flamingspork.com/projects/libeatmydata/">libeatmydata</a>. This library basically causes <code>fsck</code> calls to be ignored. Should you install <code>libeatmydata</code>? Probably not, as you might guess from the name. If you install it, you risk losing much more data if your machine crashes than you would otherwise. It may also may pose a shutdown issue, since the OS will call <code>fsck</code> before shutdown to help ensure that buffered data is written out to disk. The primary application for <code>libeatmydata</code> seems to be for OS installs. If your machine crashes during an OS install, you'd probably just re-start the install. So, why waste your time with excessive <code>fsck</code> calls?<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-34951714481352691142010-11-11T21:07:00.001-05:002010-12-12T08:27:40.613-05:00Latex: Printing Letter vs. A4<p>I love the fact that math can be typeset so easily in <a href="http://www.latex-project.org/">Latex</a>. But, I hate the fact that it can be so difficult to get it to format documents for letter paper. Previously, I've been able to edit <code>/etc/texmf/dvips/config/config.ps</code> and either move "Letter" entries to the top and/or comment-out "A4" entries. That trick didn't work this time. What did work was to skip <a href="http://en.wikipedia.org/wiki/Dvips">dvips</a> and use <a href="http://en.wikipedia.org/wiki/PdfTeX">pdflatex</a>. Somehow, it was already configured properly ("p letter" in <code>/etc/texmf/dvipdfm/config/config</code>) to use the Letter paper size. Not a problem since pdflatex works great and PDF is easy to print. I just wish someone would simplify the dvips configuration...<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-73246762883676784052010-10-29T20:17:00.001-04:002010-10-30T11:09:52.425-04:00Another Smooth Ubuntu Upgrade<p>Just upgraded my desktop Maverick Meerkat 10.10 which is a tad more complicated than my laptop because it runs apache2 to serve web pages (including 2 trac instances) and weewx to log weather information from my weather station. Thankfully, the upgrade went smoothly. When prompted for what to do with my existing apache2 config, I made a backup copy, then asked to have it overwritten. When the upgrade was complete, I uncommented the server root and copied over the trac configuration sections. After restart, everything worked as before.<br />
</p><p>The only two minor hiccups were the font (didn't look as good on the desktop as it did on the laptop) and the version of Chrome. It downgraded my chrome from 7 to 6. 'course, both of those (extremely minor) issues were easy to fix.<br />
</p><p><b>Update (10/30/10)</b>: I realized the difference in fonts. It was technically the same font (Ubuntu), but different rendering. It chose "Best shapes" on my desktop which resulted in a very skinny font. Whereas on my laptop, it chose "Subpixel smoothing". The subpixel smoothing resulted in a much thicker font. So much so that it looked like a different font!<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0tag:blogger.com,1999:blog-8225911564756141866.post-84058072971464965982010-10-26T08:06:00.003-04:002010-11-24T06:51:49.319-05:00Ubuntu Window Border Thickness<p>When I use Ubuntu on a high-resolution monitor or on a laptop (which is appx. 100% of the time), I have difficulty clicking the border of a window to resize. Apparently, <a href="https://bugs.launchpad.net/ubuntu/+source/metacity/+bug/160311">I'm not alone</a> and this has been a standing issue for years. It seems to me that it would be trivial for them to add a configuration option under "Appearance Preferences"/"Theme"/"Customize..."/"Window Border" for frame/border thickness. But, it doesn't exist yet. However, if you don't mind editing XML, the above bug describes how to make the necessary change (at least for the default, Human, theme; it looks like changing other themes would be similar). Here's the process:<br />
<ul><li>Edit the XML config: <code>sudo emacs /usr/share/themes/Human/metacity-1/metacity-theme-1.xml</code></li>
<li>Increase the values for <code>left_width</code>, <code>right_width</code> and <code>bottom_height</code>. </li>
<li>Restart the window manager (e.g. by logging out and back in)</li>
</ul>I increased the values from "3" to "5".<br />
</p><p><b>Update (10/26/10)</b>: You'll also want to increase <code>left_titlebar_edge</code> and <code>right_titlebar_edge</code> to the same value. Also, AFAICT it's not possible to change the top/titlebar border. This is a bit of a pain since I frequently try to grab the top border for resizing.<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com1tag:blogger.com,1999:blog-8225911564756141866.post-19220175509013888482010-10-25T19:27:00.000-04:002010-10-25T19:27:35.075-04:00Upgrading Ubuntu<p>Just upgraded my laptop from Lucid Lynx 10.04 to Maverick 10.10. It was (happily) largely uneventful. One thing that I had to figure out was that Upgrade Manager had been configured to only show long term support (LTS) releases, meaning it wouldn't pop-up the helpful "Upgrade" button. Once I changed "show new distribution releases" to "Normal", it showed the "Upgrade" button and the upgrade proceeded.<br />
</p><p>One touch I like in the new release is the use of round-er sans-serif fonts everywhere. Feels a bit more inviting than before. Python is 2.6.6 by default with 2.7 and 3.1 available. Unison has been upgraded to 2.32.52 which means that I'll need to upgrade my desktop before I can synchronize again.<br />
</p>Jasonhttp://www.blogger.com/profile/00489496856755184870noreply@blogger.com0