Code snippets, tech tricks and other bits and bobs

Mass delete email (or any files) over SSH

I get a lot of automated mail that I do need for analysis, but it can be a pain to delete it all.

Admittedly, if I set filters better it would be easier, but even so, if you have thousands of emails in a folder and you need to delete them using webmail remotely, it is time consuming.

So let's do it with SSH.

WARNING. This shit is dangerous. If any of this doesn't make sense DON'T DO IT!

Login to your server via SSH

Change to your mail dir (this is the location on my CentOS 5 box)
Note the mail address format - @ is escaped with \, period are replaced with underscores:

$ cd /home/{domain username}/mail/.{user\@domin_tld}/{folder}/

If you are cleaning your inbox the folder is

cur

Test the message selection before you delete!
I'm deleting Cron messages from a server. The message source for these all contain 'Subject: Cron <user\@server\>' (I've anonymised the server address)
Note the search is regex, so special chars must be escaped

$ grep -l 'Subject\: Cron \<user\@server\>' * > grep.txt

Then I open the grep.txt file, copy some filenames and then open those files to double check I'm selecting the right messages.

Once I'm satisfied I'm not selecting anything I shouldn't:

$ grep -l 'Subject\: Cron' * | xargs rm -f

Shazzam. From >3000 emails to 150 in two minutes. I should probably check that account more often...

Filed under  //   *nix   CentOS 5   Regex   SSH  

Tool causes Apache to freeze

When I read the headline, I thought they were talking about me...

http://www.h-online.com/open/news/item/Tool-causes-Apache-web-server-to-freez...

A previously unknown flaw in the code for processing byte range headers allows version 2.2.x of the Apache Web Server to be crippled from a single PC. A suitable "Apache Killer" Perl script that impressively demonstrates the problem has already been published on the Full Disclosure mailing list.

The tool sends GET requests with multiple "byte ranges" that will claim large portions of the system's memory space. A "byte range" statement allows a browser to only load certain parts of a document, for example bytes 500 to 1000. This method is used by programs such as download clients to resume downloads that have been interrupted; it is designed to reduce bandwidth requirements. However, it appears that stating multiple unsorted components in the header can cause an Apache server to malfunction.

No official patch has been released, but a functional workaround is to use rewrite rules that only allow a single range request in GET and HEAD headers. This should not present a problem for most applications. To enable the rules, administrators must load the Apache Web Server's mod_rewrite module.

Another suggested workaround is to use the mod_header module with the RequestHeader unset Range configuration to completely delete any range requests that may be contained in a header. However, this approach is likely to cause more problems than restricting the number of ranges. Admins should use the tool to test the effectiveness of their measures before others do it for them.

Filed under  //   Apache   Security   mod_rewrite  

[OS X] Set up DiffMerge for Dreamweaver file comparison

I've just started in a new office and we are all Mac'ed up, which changes the development process a bit. For one thing, I have SO much pixelated real estate I have had to put ghosts of standard window sizes on my desktop background, as it's easy to lose touch with real users' screen resolution. 1024x768 just gets lost in the corner!

One of my first productivity priorities was getting a decent visual diff tool hooked up to Dreamweaver - a colleague recommended TextWrangler but it doesn't highlight differences in clearly, so after a little research I settled on DiffMerge, a utility available for Mac, PC and *nix.

It takes a tiny bit of setting up however, a little knowledge of OS X file system, and the instructions contained a slight discrepancy in my version, so here's how if you find yourself in the same situation:

First download the DiffMerge DMG, mount the DMG (simply by double clicking the downloaded DMG) and drag DiffMerge.app to your applications.

All standard stuff so far, and any Mac user should be able to do that.

Now, launch Terminal (search for it with Finder if you need it).

Log in with your system password.

change into the mounted DMG:

$ cd /Volumes/DiffMerge\ {version number}/

For n00bs, once you have got as far as typing 'Diff' you can press [TAB] to autocomplete.

sudo copy (copy as a superuser) the shell script to /usr/bin:

$ sudo cp Extras/diffmerge.sh /usr/bin/diffmerge
 
You will be prompted for your system password

Set the permissions on the script:

$ sudo chmod 755 /usr/bin/diffmerge

Copy the man (manual file) to your system. This is where the instructions were in error as they listed the file as diffmerge.1, and my copy was diffmerge.man1:

$ sudo cp Extras/diffmerge.man1 /usr/share/man/man1/diffmerge.man1
 

Set permissions for the man file:

$ sudo chmod 644 /usr/share/man/man1/diffmerge.man1

Open Dreamweaver, and go to preferences (⌘ + U), select the 'File Compare' category and enter '{your drive}:usr:bin:diffmerge' where {your drive} is the name of your... drive... Yeh. For example: 'Macintosh HD:usr:bin:diffmerge' and click [OK]

Now you can start diffing files - when uploading and DW asks if you want to compare, click the button and DiffMerge will open automatically.

Alternatively [Ctrl] + Click on any file in choose 'Compare with {Remote Server} / {Local Server}' from the context menu as appropriate.

Filed under  //   Diff   DiffMerge   DreamWeaver   OS X   utilities  

Oops, I did it again... Outlook 2010 attachment and account check

We've all done it - please see attached.... DOH!

And with multiple email accounts, and forwarding to your main address, it's easy to send from your default account when you meant to use another...

But there are solutions to hand!

FAD (Forgotten Attachment Detection) from MS Office Labs solves problem 1:

http://www.officelabs.com/projects/forgottenattachmentdetector/Pages/default....

It requires Microsoft Visual Studio Tools for Microsoft Office V3 which is only 2mb:

http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=23656

I believe .NET is required too.

 

For account selection, a small registry modification will prompt you to choose the account each time - essentially removing the 'default account' functionality.

http://www.msoutlook.info/question/477

Filed under  //   MS Office   MS Windows   Outlook 2010   utilities  

'Deprecated' error warnings - PHP 5.3

Just reviewing some sites in my portfolio, and I notice that a CubeCart installation on a shared host is broken. Oh dear...

Deprecated: Function set_magic_quotes_runtime() is deprecated in /path/to/domain/shop/ini.inc.php on line 114
Warning: ini_set(): Cannot change zlib.output_compression - headers already sent in /path/to/domain/shop/ini.inc.php on line 118 
Warning: Cannot modify header information - headers already sent by (output started at /path/to/domain/shop/ini.inc.php:114) in /path/to/domain/shop/index_enc_ion.php on line 31 
Warning: Cannot modify header information - headers already sent by (output started at /path/to/domain/shop/ini.inc.php:114) in /path/to/domain/shop/index_enc_ion.php on line 32
Deprecated: Function eregi() is deprecated in /path/to/domain/shop/includes/functions.inc.php on line 408

Ah, great. Without warning, we've been upgraded to PHP 5.3

Thanks for that...

Anyway, it's easily solved.

With 5.3 a new error level has been introduced - E_DEPRECATED, so it's easy to suppress deprecated errors:

In your scripts:

error_reporting(E_ALL & ~E_NOTICE & ~E_DEPRECATED);

In the case of CubeCart 4 this can be acheived in ini.inc.php line 101.

In php.ini:

error_reporting = E_ALL & ~E_NOTICE & ~E_DEPRECATED

in .htaccess:

php_value error_reporting 1

.htaccess requires the correct integer value equivalent of your chosen reporting level - these are a touch obscure, but setting it to 1 means report fatal run-time errors and unrecoverable errors, which will do the trick.

Filed under  //   .htaccess   CubeCart   PHP   PHP 5.3   php.ini  

YouTube Easter Egg has Worms


A worm on YouTube, yesterday...

 This may not be news to you, but I came across a secret game of "worm" while using YouTube yesterday - while the loading icon shows, simply press any two different cursor keys and the icon zooms off and you can keep yourself entertained while waiting. Of course, should you find yourself enjoying the game and not want the video to start, you should probably be watching more interesting videos in the first place...

 

Filed under  //   Easter Egg   Game   YouTube  

Social Engine 4: Unable to delete user [Solution]

I am new to Social Engine, but have already found a number of quite surprising issues with it. As such my analysis relies upon a certain amount of logic and assumption. Ironic really, as assumptions are what cause this particular issue... 

Symptoms:

A user trying to delete their profile from http://{domain.com}/members/settings/general gets an 500 Internal Server Error.

Administrators attempting to delete profiles from admin get delete confirmation modal popup. On clicking delete, the popup turns white and user is not deleted. Inspecting traffic in the Firebug Net panel, the ajax request is returning a 500 Internal Server Error.

Checking {domain}/statistics/logs/error_log reveals the following errors generated each time:

PHP Fatal error:  Call to a member function remove() on a non-object in /path/to/domain/application/modules/Album/Model/Photo.php on line 150
PHP Fatal error:  Undefined class constant 'PRIMARY_TYPE_NUM' in /path/to/domain/application/modules/Core/Model/DbTable/Session.php on line 538

 

Cause:

I believe the cause is a user with missing images that are listed in the database. This in itself should not be an issue, but it seems the code makes fatal (!) assumptions regarding the existance of images that are listed in the database, a rookie mistake if ever there was one. When dealing with files on the file system, one must always confirm their existance, and if possible validate the file content before attempting to operate on it, else you are open to software errors and worse, security risks.

In this case the files are dealt with through objects, representing the file and the possible methods, or operations, upon it. Where a file does not exist, the object is not created, so it is the object, rather than the file whose existance needs confirmation before operating upon it.

As the source code itself notes:

    // This is dangerous, what if something throws an exception in postDelete
    // after the files are deleted?

While the author of this note is considering a different edge case, this would seem to be a worrying indicator of the standard of coding .

The 'Undefined class constant' error is merely a consequence of the prior fatal error.

The offending lines are

application/modules/Album/Model/Photo.php 
l.149 - 150

      $file = $this->api()->getApi('storage', 'storage')->get($this->file_id, null);
      $file->remove();

You could say that it is 149 that is the cause, rather than 150. 150 tries to do a perfectly valid operation upon the presumed object, but it is 149 that has failed to return an object as expected.

By that token however the cause lies in the storage API, in that it fails to return an object in the first place.

Note that exactly the same careless approach is taken immediately after this operation for thumbnails, and then again (although commented out) for cropped versions.

 

Solution:

Hack:

Add an if(is_object($object)) check before the operation:

Unix Diff:

application/modules/Album/Model/Photo.php 

150c150,152 
<       $file->remove(); 
--- 
>       if(is_object($file)){ 
>         $file->remove(); 
>       } 152c154,156 
<       $file->remove(); 
--- 
>       if(is_object($file)){ 
>         $file->remove(); 
>       }

For clarity, we are replacing:

l.149-152:

      $file = $this->api()->getApi('storage', 'storage')->get($this->file_id, null); 
      $file->remove(); 
      $file = $this->api()->getApi('storage', 'storage')->get($this->file_id, 'thumb.normal'); 
      $file->remove();

with

l.149-156:

      $file = $this->api()->getApi('storage', 'storage')->get($this->file_id, null); 
      if(is_object($file)){ 
        $file->remove(); 
      } 
      $file = $this->api()->getApi('storage', 'storage')->get($this->file_id, 'thumb.normal'); 
      if(is_object($file)){ 
        $file->remove(); 
      }

I am unsure as to just what the failed operation on 149 had returned - if it was anything equating to FALSE, then an alternative and more succinct solution might be:

      if($file = $this->api()->getApi('storage', 'storage')->get($this->file_id, null)){ 
        $file->remove(); 
      }

As this is little more than a dirty hack, I haven't taken the time to look into this further. I don't want to be creating users and deleting them just to test out a hunch. 

Proposed system solution:

The storage API should still return an object when a file is not found. This object should contain a 'not found' flag, and should not contain a remove() method, or the remove() method of said object could simply log the non existance of the file when the 'not found' flag is set.

The object could also reference a 'not found' image for use in the front end of the site in the case of images.

The advantage of this approach is that only the object definitions would need to be changed.

Alternatively, every instance of a file operation needs to have a check coded into it as per the hack above.

Further remedial action:

Having corrected this error, the system still failed to delete the user, this time throwing the following error

PHP Fatal error:  Call to a member function getNextCollectible() on a non-object in /path/to/domain/application/modules/Core/Model/Item/Collectible.php on line 54
PHP Fatal error:  Undefined class constant 'PRIMARY_TYPE_NUM' in /path/to/domain/application/modules/Core/Model/DbTable/Session.php on line 538

While I am not familiar enough with Social Engine to know exactly what collectible items are (Humbold figurines? Pokémon? I dunno...), it is the same error and can be fixed with a similar approach:

application/modules/Core/Model/Item/Collectible.php

54c54,56
<     return $this->getCollection()->getNextCollectible($this); 
--- 
>     if(is_object($this->getCollection())){ 
>       return $this->getCollection()->getNextCollectible($this); 
>     }

At this point the user could finally be deleted. 

Filed under  //   PHP   Social Engine 4  

Let them eat Pi! A truesim...

Another word for a trueism is a tautology. I thought I was quite clever for thinking of this in the context of the following article, but of course, the author had gotten there first.

Anyway, what follows is a fascinating and intelligent assasination of  π (Pi), and proposal for its replacement with τ (Tau).

http://tauday.com/

In the words of Smashing Pumpkins, "Tauday is the greatest day I've ever known"...

Filed under  //   Geometry   Mathematics   Pi   Tau  

cgi_wrapper: Per-domain php.ini under php-fastcgi (Plesk 9.5.2 + CentOS 5)

I recently discussed per-directory php directives utilising htscanner, allowing setting values with php_flag and php_value in .htaccess files.

While this is an enormous step forward in term of granular control for the webmaster, it doesn't allow complete control as some things don't work when set in .htaccess as I discovered when trying to set apc config values.

The problem stems from being unable to use vhosts.conf to specify php_admin_value as you would running PHP as an Apache module, and having to resort to other methods. I did attempt http://kb.parallels.com/9059, but it was a complete failure, because Paralells neglect to mention that you must be running Plesk 9.5.2+ in order to utilise cgi_wrapper as directed.

I had been avoiding updating Plesk, as last time I'd done so it had mangled quite a few things, but I bit the bullet, and this time the process seems to have been entirely successful.

From http://kb.parallels.com/9059

The hotfix is as follows:

It is possible to use custom php.ini for domains if php as FastCGI mode is used on it. 
It is necessary to apply hotfix. Fix is compatible with Parallels Plesk Panel versions 9.x.

Use the following instruction for applying hotfix.

1. Download hotfix from the attachment

# wget http://kb.parallels.com/Attachments/13985/Attachments/cgi_wrapper

2. Locate file cgi_wrapper

# cat /etc/psa/psa.conf |grep CGI_PHP CGI_PHP_BIN /var/www/cgi-bin/cgi_wrapper/cgi_wrapper

3. Create a backup of this file

# cp /var/www/cgi-bin/cgi_wrapper/cgi_wrapper /var/www/cgi-bin/cgi_wrapper/cgi_wrapper_orig

4. Replace cgi_wrapper with the downloaded one

# cp cgi_wrapper /var/www/cgi-bin/cgi_wrapper/cgi_wrapper

5. Copy php.ini to the configuration directory of domain

# cp php.ini /var/www/vhosts/domain.tld/conf/

6. Change permissions on the configuration directory of the domain

# chmod 0755 /var/www/vhosts/domain.tld/conf

7. Restart apache

# /etc/init.d/httpd restart
Should the download become unavailable, I have included the contents of the updated cgi_wrapper here for convenience:
#!/bin/sh

domain=`fgrep -m 1 "$UID" /etc/passwd| awk -F\: '{print $6}' |awk -F"/" '{print $5}'
PHPRC=/var/www/vhosts/$domain/conf/php.ini

[ -f ${PHPRC} ] || PHPRC="/etc/php.ini"

export PHPRC
exec /usr/bin/php-cgi -c "$PHPRC"

Filed under  //   CentOS 5   PHP   Plesk 9   Plesk 9.5.2+   cgi_wrapper   php-fastcgi   php.ini  

PHP File Upload Progress Bar

A pure PHP based (I.E. not Flash or Java based) method of monitoring file upload progress used to be something of a holy grail for web designers for some time, especially for people creating CMS or otherwise handling user provided content. I remember seeing mention of it being on the horizon many, many moons ago, but hadn't looked into it for a long time. Implementations in the CMS I have used is still very rare - where there is any it is still often Flash.

Installing APC recently, I noted the apc.rfc1867 configuration directive which states:

RFC1867 File Upload Progress hook handler is only available if APC was compiled against PHP 5.2.0 or later.

Aha, so that's the key is it? Time to investigate again methinks.

I shall at some point be taking a look at

http://www.johnboy.com/blog/a-useful-php-file-upload-progress-meter

and

http://www.johnboy.com/php-upload-progress-bar/

The second is his more recent implementation. At first glance it appears to use an iFrame - well, this is the ajax age, so I hardly think that should be an issue to bring bang up to date.

I shalll be returning to this!

Filed under  //   APC   File uploads   PHP