<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Bonuel Photography &#187; cygwin</title>
	<atom:link href="http://www.bonuel.ca/blog/tag/cygwin/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.bonuel.ca</link>
	<description>Sports &#38; Travel Photography</description>
	<lastBuildDate>Mon, 23 Jan 2012 17:19:16 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.1</generator>
		<item>
		<title>Using bash to find a file modified in a specific date/time&#160;interval?</title>
		<link>http://www.bonuel.ca/blog/2009/07/27/using-bash-to-find-a-file-modified-in-a-specific-datetime-interval/</link>
		<comments>http://www.bonuel.ca/blog/2009/07/27/using-bash-to-find-a-file-modified-in-a-specific-datetime-interval/#comments</comments>
		<pubDate>Mon, 27 Jul 2009 05:14:09 +0000</pubDate>
		<dc:creator>Miguel</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[bash]]></category>
		<category><![CDATA[computing]]></category>
		<category><![CDATA[cygwin]]></category>
		<category><![CDATA[linux]]></category>

		<guid isPermaLink="false">http://www.bonuel.ca/blog/?p=693</guid>
		<description><![CDATA[Did you ever need to find a file stored somewhere in a bunch of folders, which you didn&#8217;t remember the name, but you remember about when you last modified the file? At work, it sometimes happens that I need to find some log file, among 20 Gigs of log files, which was modified on a [...]]]></description>
			<content:encoded><![CDATA[<p>Did you ever need to find a file stored somewhere in a bunch of folders, which you didn&#8217;t remember the name, but you remember about when you last modified the file? At work, it sometimes happens that I need to find some log file, among 20 Gigs of log files, which was modified on a certain day. Here is a simple bash trick to do just that:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"> <span style="color: #c20cb9; font-weight: bold;">touch</span> <span style="color: #660033;">-d</span> <span style="color: #ff0000;">&quot;13 october 2006 15:00:00&quot;</span> ~<span style="color: #000000; font-weight: bold;">/</span>date_start
 <span style="color: #c20cb9; font-weight: bold;">touch</span> <span style="color: #660033;">-d</span> <span style="color: #ff0000;">&quot;14 october 2006 21:00:00&quot;</span> ~<span style="color: #000000; font-weight: bold;">/</span>date_end
 <span style="color: #c20cb9; font-weight: bold;">find</span> some<span style="color: #000000; font-weight: bold;">/</span>path <span style="color: #660033;">-newer</span> ~<span style="color: #000000; font-weight: bold;">/</span>date_start <span style="color: #660033;">-and</span> <span style="color: #660033;">-not</span> <span style="color: #660033;">-newer</span> ~<span style="color: #000000; font-weight: bold;">/</span>date_end</pre></div></div>

]]></content:encoded>
			<wfw:commentRss>http://www.bonuel.ca/blog/2009/07/27/using-bash-to-find-a-file-modified-in-a-specific-datetime-interval/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Start Cygwin from a folders context&#160;menu</title>
		<link>http://www.bonuel.ca/blog/2009/06/29/start-cygwin-from-a-folders-context-menu/</link>
		<comments>http://www.bonuel.ca/blog/2009/06/29/start-cygwin-from-a-folders-context-menu/#comments</comments>
		<pubDate>Tue, 30 Jun 2009 03:02:35 +0000</pubDate>
		<dc:creator>Miguel</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[bash]]></category>
		<category><![CDATA[cmd]]></category>
		<category><![CDATA[cygwin]]></category>
		<category><![CDATA[registry]]></category>
		<category><![CDATA[windows]]></category>

		<guid isPermaLink="false">http://www.bonuel.ca/blog/?p=638</guid>
		<description><![CDATA[My friend had a great little program, which essentially let you right click on any folder and open the command prompt there. The only program is that I never use the command line (CMD.EXE)&#8230; I use cygwin. A wonderful program that emulates Linux for windows, but also makes everything complicated! After a short search on [...]]]></description>
			<content:encoded><![CDATA[<p>My friend had a great little program, which essentially let you right click on any folder and open the command prompt there. The only program is that I never use the command line (CMD.EXE)&#8230; I use <a  href="http://www.cygwin.com/">cygwin</a>. A wonderful program that emulates Linux for windows, but also makes everything complicated!</p>
<p>After a short search on the web, I found a nice little discussion on a newsgroup that explained <a  href="http://sources.redhat.com/ml/cygwin/2002-05/msg01645.html">how to do it for cygwin</a>. I just wanted to note the steps for windows XP, just so I don&#8217;t forget (If you really care&#8230; you can find steps for other versions on the discussion board).</p>
<h3>1. Edit Registry</h3>
<p>Start regedit.exe.</p>

<div class="wp_syntax"><div class="code"><pre class="registry" style="font-family:monospace;">HKEY_CLASSES_ROOT\Directory\shell\BashHere=&quot;Bash Here&quot;
HKEY_CLASSES_ROOT\Directory\shell\BashHere\command=&quot;C:\\WINDOWS\\System32\\CMD.EXE /E:4096 /c C:\\cygwin\\cygwin.bat %1&quot;</pre></div></div>

<p>Adjust the path to cygwin.bat if necessary.</p>
<p>Finish regedit.</p>
<p>You should now see the entry &#8220;Bash Here&#8221; in the context menu of a folder. However, when you use it, it will start the shell in your home directory ($HOME).</p>
<h3>2. Edit cygwin.bat in c:/cygwin</h3>
<p>Insert &#8220;set BASHHERE=%1&#8243; before the line with &#8220;bash&#8230;&#8221;. This Environment variable now contains the complete path of the opened folder and is availabile in the bash shell (echo $BASHHERE).</p>
<h3>3. Edit profile</h3>
<p>Make $BASHHERE the current directory by adding the following line to<br />
/etc/profile or ~/.profile:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #000000; font-weight: bold;">if</span> <span style="color: #7a0874; font-weight: bold;">&#91;</span> <span style="color: #ff0000;">&quot;<span style="color: #007800;">$BASHHERE</span>&quot;</span> <span style="color: #000000; font-weight: bold;">!</span>= <span style="color: #ff0000;">&quot;&quot;</span> <span style="color: #7a0874; font-weight: bold;">&#93;</span>; <span style="color: #000000; font-weight: bold;">then</span>
    <span style="color: #7a0874; font-weight: bold;">cd</span> $<span style="color: #7a0874; font-weight: bold;">&#40;</span> <span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #007800;">$BASHHERE</span> <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">tr</span> <span style="color: #ff0000;">&quot;\134&quot;</span> <span style="color: #000000; font-weight: bold;">/</span><span style="color: #7a0874; font-weight: bold;">&#41;</span>
<span style="color: #000000; font-weight: bold;">fi</span></pre></div></div>

]]></content:encoded>
			<wfw:commentRss>http://www.bonuel.ca/blog/2009/06/29/start-cygwin-from-a-folders-context-menu/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Vim Tip: Restore cursor&#8217;s last&#160;position</title>
		<link>http://www.bonuel.ca/blog/2009/04/13/vim-tip-restore-cursors-last-position/</link>
		<comments>http://www.bonuel.ca/blog/2009/04/13/vim-tip-restore-cursors-last-position/#comments</comments>
		<pubDate>Tue, 14 Apr 2009 04:19:34 +0000</pubDate>
		<dc:creator>Miguel</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[bash]]></category>
		<category><![CDATA[cygwin]]></category>
		<category><![CDATA[linux]]></category>
		<category><![CDATA[tip]]></category>
		<category><![CDATA[trick]]></category>
		<category><![CDATA[unix]]></category>
		<category><![CDATA[vi]]></category>
		<category><![CDATA[vim]]></category>
		<category><![CDATA[windows]]></category>

		<guid isPermaLink="false">http://www.bonuel.ca/blog/?p=601</guid>
		<description><![CDATA[Here&#8217;s a useful vi trick that turned out to be a real time saver. It&#8217;s a set of commands you add to your vimrc file (on windows) that will recover the cursor&#8217;s last position when the file was last closed. The original tip was found on the vim wikia, which is full of those very [...]]]></description>
			<content:encoded><![CDATA[<p>Here&#8217;s a useful vi trick that turned out to be a real time saver. It&#8217;s a set of commands you add to your vimrc file (on windows) that will recover the cursor&#8217;s last position when the file was last closed. The original tip was found on the <a  href="http://vim.wikia.com/wiki/Restore_cursor_to_file_position_in_previous_editing_session">vim wikia</a>, which is full of those very useful vim tricks. Give it a try!</p>

<div class="wp_syntax"><div class="code"><pre class="vi" style="font-family:monospace;">&quot; Tell vim to remember certain things when we exit
&quot;  '10 : marks will be remembered for up to 10 previously edited files
&quot;  &quot;100 : will save up to 100 lines for each register
&quot;  :20 : up to 20 lines of command-line history will be remembered
&quot;  % : saves and restores the buffer list
&quot;  n... : where to save the viminfo files
set viminfo='10,\&quot;100,:20,%,n~/.viminfo
&nbsp;
&quot; when we reload, tell vim to restore the cursor to the saved position
augroup JumpCursorOnEdit
 au!
 autocmd BufReadPost *
 \ if expand(&quot;&lt;afile&gt;:p:h&quot;) !=? $TEMP |
 \ if line(&quot;'\&quot;&quot;) &gt; 1 &amp;&amp; line(&quot;'\&quot;&quot;) &lt;= line(&quot;$&quot;) |
 \ let JumpCursorOnEdit_foo = line(&quot;'\&quot;&quot;) |
 \ let b:doopenfold = 1 |
 \ if (foldlevel(JumpCursorOnEdit_foo) &gt; foldlevel(JumpCursorOnEdit_foo - 1)) |
 \ let JumpCursorOnEdit_foo = JumpCursorOnEdit_foo - 1 |
 \ let b:doopenfold = 2 |
 \ endif |
 \ exe JumpCursorOnEdit_foo |
 \ endif |
 \ endif
 &quot; Need to postpone using &quot;zv&quot; until after reading the modelines.
 autocmd BufWinEnter *
 \ if exists(&quot;b:doopenfold&quot;) |
 \ exe &quot;normal zv&quot; |
 \ if(b:doopenfold &gt; 1) |
 \ exe &quot;+&quot;.1 |
 \ endif |
 \ unlet b:doopenfold |
 \ endif
augroup END</pre></div></div>

]]></content:encoded>
			<wfw:commentRss>http://www.bonuel.ca/blog/2009/04/13/vim-tip-restore-cursors-last-position/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Fixing ASCII/Text to Binary mode in&#160;CVS</title>
		<link>http://www.bonuel.ca/blog/2008/12/20/fixing-asciitext-to-binary-mode-in-cvs/</link>
		<comments>http://www.bonuel.ca/blog/2008/12/20/fixing-asciitext-to-binary-mode-in-cvs/#comments</comments>
		<pubDate>Sun, 21 Dec 2008 03:35:17 +0000</pubDate>
		<dc:creator>Miguel</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[bash]]></category>
		<category><![CDATA[binary]]></category>
		<category><![CDATA[cvs]]></category>
		<category><![CDATA[cygwin]]></category>
		<category><![CDATA[files]]></category>
		<category><![CDATA[linux]]></category>

		<guid isPermaLink="false">http://www.bonuel.ca/blog/2008/12/20/fixing-asciitext-to-binary-mode-in-cvs/</guid>
		<description><![CDATA[I&#8217;ve often found myself in the situation where I need to quickly fix the file mode from ASCII to Binary in CVS. Here are quick instructions: cvs admin -kb BinaryFile.xls cvs update -A BinaryFile.xls cvs commit -m &#34;make it binary&#34; BinaryFile.xls]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve often found myself in the situation where I need to quickly fix the file mode from ASCII to Binary in CVS. Here are quick instructions:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #c20cb9; font-weight: bold;">cvs</span> admin <span style="color: #660033;">-kb</span> BinaryFile.xls
<span style="color: #c20cb9; font-weight: bold;">cvs</span> update <span style="color: #660033;">-A</span> BinaryFile.xls
<span style="color: #c20cb9; font-weight: bold;">cvs</span> commit <span style="color: #660033;">-m</span> <span style="color: #ff0000;">&quot;make it binary&quot;</span> BinaryFile.xls</pre></div></div>

]]></content:encoded>
			<wfw:commentRss>http://www.bonuel.ca/blog/2008/12/20/fixing-asciitext-to-binary-mode-in-cvs/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Useful keyboard shortcuts for&#160;Bash</title>
		<link>http://www.bonuel.ca/blog/2008/12/16/useful-keyboard-shortcuts-for-bash/</link>
		<comments>http://www.bonuel.ca/blog/2008/12/16/useful-keyboard-shortcuts-for-bash/#comments</comments>
		<pubDate>Tue, 16 Dec 2008 05:48:47 +0000</pubDate>
		<dc:creator>Miguel</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[bash]]></category>
		<category><![CDATA[cygwin]]></category>
		<category><![CDATA[keyboard]]></category>
		<category><![CDATA[linux]]></category>
		<category><![CDATA[shortcuts]]></category>

		<guid isPermaLink="false">http://www.bonuel.ca/blog/?p=379</guid>
		<description><![CDATA[I&#8217;ve been playing around with bash for the past two years now, and found some of these keyboard shortcuts to be very useful. I&#8217;ve tried to put the ones I use the most often on here. For other cool bash shortcuts and tricks, visit CTRL + R: Reverse search through history for a previously used [...]]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve been playing around with bash for the past two years now, and found some of these keyboard shortcuts to be very useful. I&#8217;ve tried to put the ones I use the most often on here. For other cool bash shortcuts and tricks, visit </p>
<ul>
<li><strong>CTRL + R</strong>: Reverse search through history for a previously used command;</li>
<li><strong>CTRL + A</strong>: Moves your cursor to the beginning of the current line;</li>
<li><strong>CTRL + E</strong>: Moves your cursor to the end of the current line;</li>
<li><strong>ALT + F</strong>: Moves your cursor forward one word on the current line;</li>
<li><strong>ALT + B</strong>: Moves your cursor backward one word on the current line;</li>
<li><strong>CTRL + W</strong>: Deletes the word before the cursor;</li>
<li><strong>ESC + D</strong>: Deletes the word after the cursor;</li>
<li><strong>CTRL + U</strong>: Clears the line before the cursor position. If you are at the end of the line, clears the entire line;</li>
<li><strong>CTRL + K</strong>: Clears the line after the cursor;</li>
<li><strong>CTRL + C</strong>: Kills the current running process;</li>
<li><strong>CTRL + Z</strong>: Suspends the current job and runs it as a background process;</li>
<li><strong>TAB</strong>: Auto-complete files and folder names;</li>
</ul>
<p>For the sake of completeness, I&#8217;ve added these other shortcuts that I saw on another website. However, I find that I do not use those on a regular basis. In reality, there are plenty more bash keyboard shortcuts around, but I just never bothered trying them out yet&#8230; For a much more extensive list of shortcuts (way too extensive for my tastes), visit <a  href="http://blog.webhosting.uk.com/web-hosting/using-bash-shell-shortcuts/">web hosting uk</a>&#8216;s blog post on the subject.</p>
<ul>
<li><strong>CTRL + L</strong>: Clears the Screen (I use the &#8220;clear&#8221; command);</li>
<li><strong>CTRL + H</strong>: Same as backspace;</li>
<li><strong>CTRL + D</strong>: Exit the current shell (I type &#8220;exit&#8221; in Putty);</li>
<li><strong>CTRL + T</strong>: Swaps the last two characters before the cursor;</li>
<li><strong>ESC + T</strong>: Swaps the last two words before the cursor;</li>
<li><strong>CTRL + XX</strong>: Moves between end-of-line and current cursor position;</li>
<li><strong>CTRL + Y</strong>: Recovers previous deletion (Not file deletion!);</li>
</ul>
]]></content:encoded>
			<wfw:commentRss>http://www.bonuel.ca/blog/2008/12/16/useful-keyboard-shortcuts-for-bash/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>grep through gzip files vs tarball&#160;files</title>
		<link>http://www.bonuel.ca/blog/2008/11/20/grep-through-gzip-files-vs-tarball-files/</link>
		<comments>http://www.bonuel.ca/blog/2008/11/20/grep-through-gzip-files-vs-tarball-files/#comments</comments>
		<pubDate>Thu, 20 Nov 2008 05:31:13 +0000</pubDate>
		<dc:creator>Miguel</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[bash]]></category>
		<category><![CDATA[cygwin]]></category>
		<category><![CDATA[grep]]></category>
		<category><![CDATA[gzip]]></category>
		<category><![CDATA[linux]]></category>
		<category><![CDATA[space]]></category>
		<category><![CDATA[tar]]></category>
		<category><![CDATA[tarball]]></category>
		<category><![CDATA[xargs]]></category>

		<guid isPermaLink="false">http://www.bonuel.ca/blog/?p=357</guid>
		<description><![CDATA[I recently came across a couple of instances, on random forums, where the user was trying to grep through a huge log file that had been tarred and then gzipped. Only then did I realized that some people out there do not know the differences and advantages between a gzip file and a tarball (actually, [...]]]></description>
			<content:encoded><![CDATA[<p>I recently came across a couple of instances, on random forums, where the user was trying to grep through a huge log file that had been tarred and then gzipped. Only then did I realized that some people out there do not know the differences and advantages between a gzip file and a tarball (actually, that particular dude didn&#8217;t know purpose of a tarball). So, for my personal benefit, I did a couple of experiments with some big log files (413 Megabytes of text file) to see what the speed vs storage benefits were when using gzip and tar.gz, all with bash (sorry tcsh users!).</p>
<h3>A little on gzip files</h3>
<p>A gzip file is simply a file formatted in a certain way, using a compression algorithm called DEFLATE, that makes the file smaller (<a  href="http://en.wikipedia.org/wiki/Gzip_compression">read more on wikipedia</a>) that It is best to use gzip when trying to save some space, but still want to have easy access to your files for peeking in them (see speed results of grepping through log files at bottom). The scenario where I use gzip the most is for compressing 100-200 log files, each 20-30 megs (again, see stats at bottom for space gains). An important feature of gzip is that it applies to single file (this is not a ZIP file people!) and so you will end up with 100-200 little gzip files, instead of big ones. This can be a little annoying for transferring to clients, and handling in general (notice the for loop in my bash script).</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #666666; font-style: italic;"># gzip multiple files</span>
<span style="color: #000000; font-weight: bold;">for</span> i <span style="color: #000000; font-weight: bold;">in</span> <span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">find</span> . <span style="color: #660033;">-name</span> <span style="color: #ff0000;">&quot;*.log&quot;</span><span style="color: #000000; font-weight: bold;">`</span>; <span style="color: #000000; font-weight: bold;">do</span> <span style="color: #c20cb9; font-weight: bold;">gzip</span> <span style="color: #007800;">$i</span>; <span style="color: #000000; font-weight: bold;">done</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># grep through all those gzip files</span>
<span style="color: #000000; font-weight: bold;">for</span> i <span style="color: #000000; font-weight: bold;">in</span> <span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">find</span> . <span style="color: #660033;">-name</span> <span style="color: #ff0000;">&quot;*.log.gz&quot;</span><span style="color: #000000; font-weight: bold;">`</span>;<span style="color: #000000; font-weight: bold;">do</span> <span style="color: #c20cb9; font-weight: bold;">zgrep</span> <span style="color: #ff0000;">&quot;TIME&quot;</span> <span style="color: #007800;">$i</span>; <span style="color: #000000; font-weight: bold;">done</span></pre></div></div>

<h3>A little on tarball files</h3>
<p>The ideal situation in which to use a tarball is when you want to compress directories and data files, which you want to bundle up in one nice and tidy package for users/clients to download but still preserve some of the file system information such as permissions and directory structure (<a  href="http://en.wikipedia.org/wiki/Tarball">read more on wikipedia</a>). Note here that the end product is one file, which can be <strong>extremely useful</strong> in some cases (even a necessity at times). I say a necessity because sometimes, when handing in log files through a ftp, you may want to have one package that you&#8217;ve encrypted for your client (using pgp), in which case you wouldn&#8217;t want to have 20-30 small packages all encrypted separately&#8230; In any case, to produce a tarball with a lot of logs, you can do the following under bash:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #666666; font-style: italic;"># Create a tarball</span>
<span style="color: #c20cb9; font-weight: bold;">tar</span> <span style="color: #660033;">-czvf</span> logs.tar.gz <span style="color: #000000; font-weight: bold;">*</span>.log
&nbsp;
<span style="color: #666666; font-style: italic;"># grep through a tarball</span>
<span style="color: #c20cb9; font-weight: bold;">zcat</span> logs.tar.gz <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">tar</span> <span style="color: #660033;">-xvf</span> - <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">xargs</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> <span style="color: #ff0000;">&quot;TIME&quot;</span></pre></div></div>

<h3>Storage vs Speed for grepping through different types of files</h3>
<table summary="Some statistics on storage vs speed, while grepping in different types of files."  border="0" cellspacing="0" cellpadding="0">
<caption><em>Some statistics on storage vs speed, while grepping in different types of files.</em></caption>
<tr>
<th class="span-4">Type of file</th>
<th class="span-4 last">Size of files (Megabytes)</th>
<th class="span-4 last">Time for grep (Seconds)</th>
</tr>
<tr>
<td>RAW</td>
<td>413 248</td>
<td>2</td>
</tr>
<tr>
<td>GZIP</td>
<td>27 656</td>
<td>3</td>
</tr>
<tr>
<td>TARBALL</td>
<td>27 520</td>
<td>19</td>
</tr>
</table>
<p><strong>Notice enormous gain in space going from raw log files to gzip log files</strong>. You have a <em>93% reduction in size</em> (RAW to GZIP), compared to a mere 0.05% reduction in size by going from many GZIP files to one tarball. Now, I didn&#8217;t even talk about the loss in speed when compressing. That is, of course, the most important thing to consider when dealing with files that you will need to peek through from time to time (logs are a perfect example). </p>
<p><a  href="http://www.bonuel.ca/blog/wp-content/uploads/2008/11/chart.png"><img src="http://www.bonuel.ca/blog/wp-content/uploads/2008/11/chart.png" alt="" title="Stats for grep through gzip and tar.gz files" width="180" height="120" class="alignright size-medium wp-image-359" /></a> In order to compare the speed for each scenario, i used a very simple bash script, which I copied here for documentation (notice that I redirect the output to a &#8216;toto&#8217; file so that i don&#8217;t get anything printed on my screen). The performance of my &#8216;grep&#8217; command on the RAW logs was very good, 2 seconds to find 107 336 occurences of &#8220;TIME&#8221; in the 10 logs. Now comparing this with the results of the GZIP logs and the TARBALL, 3 and 19 respectively, you can quickly see that <em>it is extremely advantageous to use gzip for log files</em> (Look at the little graph of the different time if you are a more visual person&#8230;). Not only the gain in storage is negligible when going from GZIP to TARBALL, but the speed at which you have access to your data is a lot slower.</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #ff0000;">&quot;Testing speed of RAW&quot;</span>
<span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #ff0000;">&quot;====================&quot;</span>
<span style="color: #7a0874; font-weight: bold;">echo</span> $<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">date</span><span style="color: #7a0874; font-weight: bold;">&#41;</span>
<span style="color: #000000; font-weight: bold;">for</span> i <span style="color: #000000; font-weight: bold;">in</span> <span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">find</span> . <span style="color: #660033;">-name</span> <span style="color: #ff0000;">&quot;*.log&quot;</span><span style="color: #000000; font-weight: bold;">`</span>;<span style="color: #000000; font-weight: bold;">do</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> <span style="color: #ff0000;">&quot;TIME&quot;</span> <span style="color: #007800;">$i</span> <span style="color: #000000; font-weight: bold;">&gt;&gt;</span> toto; <span style="color: #000000; font-weight: bold;">done</span>
<span style="color: #7a0874; font-weight: bold;">echo</span> $<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">date</span><span style="color: #7a0874; font-weight: bold;">&#41;</span>
&nbsp;
<span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #ff0000;">&quot;Testing speed of GZIP&quot;</span>
<span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #ff0000;">&quot;====================&quot;</span>
<span style="color: #7a0874; font-weight: bold;">echo</span> $<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">date</span><span style="color: #7a0874; font-weight: bold;">&#41;</span>
<span style="color: #000000; font-weight: bold;">for</span> i <span style="color: #000000; font-weight: bold;">in</span> <span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">find</span> . <span style="color: #660033;">-name</span> <span style="color: #ff0000;">&quot;*.log.gz&quot;</span><span style="color: #000000; font-weight: bold;">`</span>;<span style="color: #000000; font-weight: bold;">do</span> <span style="color: #c20cb9; font-weight: bold;">zgrep</span> <span style="color: #ff0000;">&quot;TIME&quot;</span> <span style="color: #007800;">$i</span> <span style="color: #000000; font-weight: bold;">&gt;&gt;</span> toto; <span style="color: #000000; font-weight: bold;">done</span>
<span style="color: #7a0874; font-weight: bold;">echo</span> $<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">date</span><span style="color: #7a0874; font-weight: bold;">&#41;</span>
&nbsp;
<span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #ff0000;">&quot;Testing speed of TARBALL&quot;</span>
<span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #ff0000;">&quot;=======================&quot;</span>
<span style="color: #7a0874; font-weight: bold;">echo</span> $<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">date</span><span style="color: #7a0874; font-weight: bold;">&#41;</span>
<span style="color: #c20cb9; font-weight: bold;">zcat</span> logs.tar.gz <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">tar</span> <span style="color: #660033;">-xvf</span> - <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">xargs</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> <span style="color: #ff0000;">&quot;TIME&quot;</span> <span style="color: #000000; font-weight: bold;">&gt;&gt;</span> toto
<span style="color: #7a0874; font-weight: bold;">echo</span> $<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">date</span><span style="color: #7a0874; font-weight: bold;">&#41;</span></pre></div></div>

<h3>Concluding remark</h3>
<p>The are situations where a tarball is necessary (or advantageous), but, in general, <strong>to keep the size of many log files down and still be able to search through them, I recommend using gzip</strong>. Not to mention that all your favorite bash commands come in a gzip flavour (zcat, zgrep, zdiff, zmore, etc) and vi can easily read a gzip file on the fly! What more can you ask for!</p>
]]></content:encoded>
			<wfw:commentRss>http://www.bonuel.ca/blog/2008/11/20/grep-through-gzip-files-vs-tarball-files/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Emulating the Recycling Bin in&#160;bash</title>
		<link>http://www.bonuel.ca/blog/2008/01/17/emulating-the-recycling-bin-in-bash/</link>
		<comments>http://www.bonuel.ca/blog/2008/01/17/emulating-the-recycling-bin-in-bash/#comments</comments>
		<pubDate>Thu, 17 Jan 2008 18:58:28 +0000</pubDate>
		<dc:creator>Miguel</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[bash]]></category>
		<category><![CDATA[cygwin]]></category>
		<category><![CDATA[linux]]></category>

		<guid isPermaLink="false">http://www.bonuel.ca/blog/2008/01/17/emulating-the-recycling-bin-in-bash/</guid>
		<description><![CDATA[I just ran into a problem where I accidentally erased a file that was very important and could not recover from it (it was a monstrous statistical model that takes about 5 hours to re-train…). I decided I had enough of this and needed a way to be able to recover from those silly mistakes [...]]]></description>
			<content:encoded><![CDATA[<p>I just ran into a problem where I accidentally erased a file that was very important and could not recover from it (it was a monstrous statistical model that takes about 5 hours to re-train…). I decided I had enough of this and needed a way to be able to recover from those silly mistakes (apparently not making them in the first place just wasn’t an option…)  I wondered if anybody ever run into this problem, and what was their personal solution? Here is what I found and tested today (it works pretty well).</p>
<p>I know that there are processes that backs up your whole $HOME on a regular schedule, I forget the name of this process, but that&#8217;s not exactly what I need. I always liked the idea of a “recycling bin” from the GUI Operating Systems like Windows (or KDE) and decided that that&#8217;s exactly what I needed. I found the next best thing on the internet (See <a  href="http://albatross.dnsdojo.net/apache2-default/wiki/index.php/Bash_Tools">John&#8217;s Bash Tools</a> for his original post).</p>
<p>I simply put this in my .bashrc and now can use “junk” instead of “rm”… It’s a pretty neat little trick actually. I then can use “empty_junk” every now and then to keep the size of my home reasonable. I could even add &#8220;empty_junk&#8221; in my .bashrc so that it gets emptied everytime I log on (I won&#8217;t because I often log onto twice at the same time and would create problems&#8230;).</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #666666; font-style: italic;">##############################</span>
<span style="color: #666666; font-style: italic;"># Emulating a reclycling bin #</span>
<span style="color: #666666; font-style: italic;">##############################</span>
<span style="color: #666666; font-style: italic;"># This function creates a directory called .junk</span>
<span style="color: #666666; font-style: italic;"># and creates directories labeled by the date</span>
<span style="color: #666666; font-style: italic;"># inside of this directory. Then, when a file is</span>
<span style="color: #666666; font-style: italic;"># &quot;junk&quot;ed, it is actually just moved to this</span>
<span style="color: #666666; font-style: italic;"># directory. The command line accepts as many</span>
<span style="color: #666666; font-style: italic;"># files as you would care to junk, and can even</span>
<span style="color: #666666; font-style: italic;"># work on entire directories. Be careful, since it</span>
<span style="color: #666666; font-style: italic;"># is really just a wrapped &quot;mv&quot; command, you may</span>
<span style="color: #666666; font-style: italic;"># overwrite your junked files if they both have</span>
<span style="color: #666666; font-style: italic;"># the same name and end up in the root directory.</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">function</span> junk <span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #7a0874; font-weight: bold;">&#41;</span> <span style="color: #7a0874; font-weight: bold;">&#123;</span>
    <span style="color: #007800;">TODAY</span>=$<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #c20cb9; font-weight: bold;">date</span> +<span style="color: #000000; font-weight: bold;">%</span>d-<span style="color: #000000; font-weight: bold;">%</span>m-<span style="color: #000000; font-weight: bold;">%</span>y<span style="color: #7a0874; font-weight: bold;">&#41;</span>
    <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #7a0874; font-weight: bold;">&#91;</span> <span style="color: #000000; font-weight: bold;">!</span> <span style="color: #660033;">-e</span> <span style="color: #007800;">$HOME</span><span style="color: #000000; font-weight: bold;">/</span>.junk<span style="color: #000000; font-weight: bold;">/</span><span style="color: #007800;">$TODAY</span> <span style="color: #7a0874; font-weight: bold;">&#93;</span>
    <span style="color: #000000; font-weight: bold;">then</span>
        <span style="color: #c20cb9; font-weight: bold;">mkdir</span> <span style="color: #660033;">-p</span> <span style="color: #007800;">$HOME</span><span style="color: #000000; font-weight: bold;">/</span>.junk<span style="color: #000000; font-weight: bold;">/</span><span style="color: #007800;">$TODAY</span>
    <span style="color: #000000; font-weight: bold;">fi</span>
&nbsp;
    <span style="color: #000000; font-weight: bold;">for</span> x <span style="color: #000000; font-weight: bold;">in</span> $<span style="color: #000000; font-weight: bold;">@</span>
    <span style="color: #000000; font-weight: bold;">do</span>
        <span style="color: #c20cb9; font-weight: bold;">mv</span> <span style="color: #007800;">$x</span> <span style="color: #007800;">$HOME</span><span style="color: #000000; font-weight: bold;">/</span>.junk<span style="color: #000000; font-weight: bold;">/</span><span style="color: #007800;">$TODAY</span>
    <span style="color: #000000; font-weight: bold;">done</span>
<span style="color: #7a0874; font-weight: bold;">&#125;</span>
&nbsp;
<span style="color: #666666; font-style: italic;"># A very simple function to empty the trash.</span>
<span style="color: #666666; font-style: italic;"># I set -Ri so that it recurses but prompts you.</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">function</span> empty_junk <span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #7a0874; font-weight: bold;">&#41;</span> <span style="color: #7a0874; font-weight: bold;">&#123;</span>
    <span style="color: #c20cb9; font-weight: bold;">rm</span> <span style="color: #660033;">-Ri</span> <span style="color: #007800;">$HOME</span><span style="color: #000000; font-weight: bold;">/</span>.junk<span style="color: #000000; font-weight: bold;">/*</span>
<span style="color: #7a0874; font-weight: bold;">&#125;</span></pre></div></div>

]]></content:encoded>
			<wfw:commentRss>http://www.bonuel.ca/blog/2008/01/17/emulating-the-recycling-bin-in-bash/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

