Building Useful Agents 01

December 12, 2016 Leave a comment

This is a POC. Hunting through multiple pages of content while sites throw up multiple random pages triggered by javascript is very annoying. This useful agent requires a file called show_list.txt including PERL regular expressions if you are so inclined. It’s a basic list of shows you want it to find.

3 Percent
^Aftermath S0
Agent X
Aquarius (US)*
Ash vs Evil Dead
Better Call Saul
Blindspot S0
DC[s]* Legends
Elementary S0

The file is read into the script and all the spaces are replaced by \W* to cope with . _ etc.

This script will load and parse the website looking for words matching the expressions and create a HTML page with links.

The last part of the script also looks for the latest version of the file I currently have and then I can decide whether I need a new version or not.

Now I can run this agent to the page depth I define and it will return matches based on the shows in show_list.txt

At a page depth of 1 it runs in about 5 seconds, at a page depth of 4 it runs in about 10 seconds. Which is so much more efficient than manually checking those pages myself.

This can be be extended (and you can also use other tools that have been around for a long time) however I find that I usually learn more building it myself.

#useful sites:
# Multi line grep
# command: pcregrep -M '<tr name="hover" class="forum_header_border">.*(\n|.)*?</tr>' homepage.txt
# Helpful site for regex

if [ $# -eq 0 ]; then
echo "No arguements supplied - defaulting for 4 pages"

echo $PAGES

while IFS='' read -r line || [[ -n "$line" ]]; do
echo -n "Text read from file: $line"
SHOW_LIST1=$(echo $line | sed 's_[.\ ]_\\W*_g')
echo " --> $SHOW_LIST1"
if [ $COUNT -eq 0 ]; then
# echo $SHOW_LIST

((COUNT ))
done < "show_list.txt"

for i in $(seq 0 $PAGES);
if [ $i -eq 0 ]; then
RESPONSE=$(curl -s ";)
RESULT=$(echo "$RESPONSE" | grep -iP "$SHOWS" | grep -iP "download|magnet" )
echo result="$RESULT"
echo -------------

echo $URL1
echo $PAGE
RESPONSE=$(curl -s $URL1)
#RESULT1=$(echo "$RESPONSE" | grep -iP "$SHOWS" | grep -i download | grep -v 720p)
RESULT1=$(echo "$RESPONSE" | grep -iP "$SHOWS" | grep -iP "download|magnet")
echo result1="$RESULT1"
echo \n result="$RESULT"
echo -------------

echo ****** FINAL ******

echo "$RESULT"
echo "$RESULT" > result.txt

OUTPUT=$(echo "$RESULT" | sed s/title=\"/\>/g | sed s/\"\>\</\</g | sed s/a\>/a\>\<p\>/g)
echo "$(date) <BR>PAGES=$PAGES<P>" > _show_list.html
echo $OUTPUT >> _show_list.html

---- you will need to customise this for your system -----
echo "<P><P> File List <P><P>" >> _show_list.html
echo "<table border=0>" >> _show_list.html

FILES=$(find /data01/download/complete/ -maxdepth 5 -printf "%f\n")
#echo "$FILES"

for f in $(echo $SHOW_LIST | sed 's/|/ /g')
NICE_FILE=$(echo $f | sed 's/\\W\*/ /g')
echo "<TR><TD>" >> _show_list.html
echo -n "$NICE_FILE</TD>" >> _show_list.html
FILE_NAME=$(echo "$FILES" | sed 's/[.-]/ /g' | grep -iP "$f" | sort -rf | head -1)
echo "<TD> $FILE_NAME </TD></TR>" >> _show_list.html

echo "</table>" >> _show_list.html
#echo "<P><P><P>ALL FIELS<P><P>$FILES" >> _show_list.html

cp _show_list.html /data01/download/complete/_showlist.html

echo "**** COMPLETE ****"

Categories: Technology Tags: , ,

Stop almost all YouTube and internet ads in less than 30 minutes using a Squid Proxy Server

October 17, 2015 Leave a comment

no andsI use Adblock or Adblock Plus in all my web browsers so I hardly ever see an ad but when I’m using the web however when I (or the family) use Apple, Android or XBox devices to watch youtube content online or browse the internet I’m constantly reminded how annoying ads actually are.

Someone in my family (not me) was trying to watch Channel 10 catchup TV today day and before they could start watching the show they had to sit through 2 ads and then there were more ads (up to 4-5 ads at a time) throughout the broadcast as well.

That is a rubbish experience & I knew I could make it better.

I’d been putting it off for a long while – mostly due to time constraints – but when I actually focused on enabling ad blocking via my Squid proxy it took less than 30 minutes to block approx 95% of all ads supported by adblock plus including youtube, google, etc and for right now – my family is happy that they can watch Channel 10 catchup TV without any ads & with no other counter measures in place approx 95% of all web based ads (including youtube ads on apple devices etc)  have been blocked on local apple devices using the proxy.

If (like me) you already have a Squid server running – I highly recommend – – just make sure you put their recommendations at the top of your squid.conf file – otherwise it didn’t work for me.  I also had to add a few additional domains to the ad_block.txt file to block some of the local advertising.

The main reason I’m excited about this is that I can apply a single rule to ignore a new ad server and have it apply across all the devices on my local network that didn’t have any adblocking capability previously.

I’m also aware that there are many ways to circumvent this this type of blocking – however – until then – everyone on my lan is enjoying ad free browsing.

I’m thinking about proving a Raspberry Pi turnkey solution as a side project, would you be interested?

Categories: General, Technology Tags: ,

Hard Disk Drive Data Recovery Using A Freezer Actually Works

February 4, 2015 1 comment

Frozen HDDAfter a Linux kernel upgrade, an old data hard drive in one of my Linux boxes (which I very rarely reboot) failed last night so the box  wouldn’t boot without manually skipping the HDD.  Recovering the Linux box simply meant removing the old data drive, swapping it with another drive I had lying around, updating the drive UUID in /etc/fstab and away the server went again.  Plus the server got a 320GB data drive upgrade in the process 🙂

Loosing the HDD data drive wasn’t a really big issue as the drive was automatically backed up about a week ago and it only stores media files and other transient data so it wasn’t a huge loss if I couldn’t recover it however, it would be much easier if I could get the last week of data off the drive.

Plugging the problem drive into an external HDD case and trying to mount it on another machine was no good.  I could hear the drive spinning up and the hdd heads trying to engage with the disk platters but it just wouldn’t start.  Then, I remembered a very old (but fairly well known trick) which is to put the HDD in the Freezer for an hour or so and then retry accessing the drive.  I’d never had the opportunity (or reason) to try this before and I though it was something worth testing so I put the HDD in a sealed zip lock plastic bag and stuck it in the freezer.

After an hour I removed the HDD from the zip lock, plugged it back into the external HDD case and connected it to my PC.  I fired up a System Rescue Linux VM and attempted to attach the HDD.  I could hear the drive spin up and the heads were now engaging with the disk platters, VMWare reported that the drive was attached and available but the Linux VM wouldn’t read the drive (fdisk -l just hung).  After about 5 minutes and a few restarts the drive it was finally recognised by Linux and I was able to copy the data off the drive.

I must admit, I’d heard the rumour that freezing a hard drive would allow it to work for long enough to copy the data off it many times before but now that I’ve actually tested it myself I can confirm that it does actually work.

Final thoughts, even though I had a backup it was still a week old, make sure you backup your data regularly as you never know when a drive might fail and also ensure you backup any important data BEFORE a server restart.

Categories: Technology Tags: , , , ,

Xbox Live Status Script

December 27, 2014 Leave a comment

If you’re like me, frustrated by all the Xbox One DDOS going on over the Christmas Holidays and have a linux box lying around here is a small script to check the service status in Australia.

Just cron it and output it to a log file – after that you can do what ever you want with the output.

I hooked the output up to my Raspberry Pi and changed the color of an LED  from Red to Green based on the Xbox Live Core Services Status.

# Author: Merric Reese
# Date: Dec 26, 2014
# Purpose: Show Xbox Live Status and report when Core Services are online
# Notes: Common sense rules apply to usage - I only run this script about every 5 minutes during a service outage 
MR_PAGE=$(curl -s
MR_XBOX_CORE=$(echo "$MR_PAGE" |grep -Eo "Xbox Live Core Services.*[L|l]imited|Xbox Live Core Services.*[A|a]ctive" |grep -iEo "active|limited")
MR_XBOX_PURCHASES=$(echo "$MR_PAGE" |grep -Eo "Purchase and Content Usage.*[L|l]imited|Purchase and Content Usage.*[A|a]ctive" |grep -iEo "active|limited")
MR_XBOX_GAMING=$(echo "$MR_PAGE" |grep -Eo "Social and Gaming.*[L|l]imited|Social and Gaming.*[A|a]ctive" |grep -iEo "active|limited")
MR_XBOX_WEBSITE=$(echo "$MR_PAGE" |grep -Eo "<h3>Website.*[L|l]imited.*</h3>|<h3>Website.*[A|a]ctive.*</h3>" |grep -iEo "active|limited")
MR_XBOX_TV=$(echo "$MR_PAGE" |grep -Eo "TV, Music and Video.*[L|l]imited|TV, Music and Video.*[A|a]ctive" |grep -iEo "active|limited")
if [ "$MR_XBOX_CORE" != "Limited" ] && [ "$MR_XBOX_GAMING" != "Limited" ]; then
# echo "#"
echo "### FIXED ### " $(date) "Xbox Live Core Services: $MR_XBOX_CORE | Social and Gaming: $MR_XBOX_GAMING | Purchases and Content: $MR_XBOX_PURCHASES | Website: $MR_XBOX_WEBSITE | TV, Music and Video: $MR_XBOX_WEBSITE"
# echo "#"
echo "waiting... " $(date) "Xbox Live Core Services: $MR_XBOX_CORE | Social and Gaming: $MR_XBOX_GAMING | Purchases and Content: $MR_XBOX_PURCHASES | Website: $MR_XBOX_WEBSITE | TV, Music and Video: $MR_XBOX_WEBSITE"

BTW: I haven’t spent the whole time coding or waiting for Xbox Live to come back online.  I’d just like to able to download the free games when the service is available and this was an easier way to check the service status.

If you found this useful then please leave a comment.

Have a very Merry Christmas and a Happy New Year.

Categories: Technology Tags: , , ,

Panasonic PT-AE900E fan noise is actually the Dynamic Iris

July 5, 2014 4 comments


My Panasonic PT-AE900E projector (same as PT-AE900U) is getting a bit long in the tooth and started making a noise that sounded like a PC power supply fan that is about to die (or a couple of marbles rolling around the bottom of the projector).  The sound appears as soon as the projector is turned on and remains there until the project is switched off.

I pulled the projector apart (not for the faint-heared) to see if I could find the fan that was about to cease up but they all seemed fine.

After some additional research on Google I found this AVS forum discussing a similar noise and it suggested disabling the dynamic iris as that could be the cause of the issue.  I eventually found the PT-AE900E Service Manual (thanks to Garry Tan’s blog post).

Fortunately, I also found this Youtube Video on a fix for the dynamic iris an older model Panasonic projector.  The iris control is a little bit harder to get to on the AE900E but the iris control is accessed through the lamp cover the same way.

The video suggests using CRC or WD40 on a cotton bud but I chose to use some sewing machine oil.

When I first moved the iris it took about 1/2 a second to go from fully open to fully closed.  After lubricating the iris shaft and spring (as per the video) it now shuts in a couple of milliseconds.

After closing up the projector and testing it the noise is completely gone & the dynamic iris is now working as expected.

I also found this Spare Parts suppler in Australia in case you need a replacement iris.

Usual warnings – Make sure the projector is unplugged before performing any of the above, be extremely careful and don’t try this unless your confident you know what you are doing.


Kubuntu 12.04 LTS on a Dell T1600 with Nvidia Card

July 17, 2012 1 comment

I’ve been trying to install Kubuntu 12.04 LTS on a Dell T1600 for about 45 minutes now.  Each time it boots off the CD it ends up on a blank screen with a cursor flashing in the top left corner of the screen.  The install has obviously hung but there is no feedback. I also burnt a new CD incase it was the issue but still no joy.

So, as always, Google is my friend and I’ve found several other people (even with different distros) having the same problem and it appears to be related to the nvidia card in the system and the open source drivers.  Using the open source nouveau driver will cause the install to hang.  There is also some threads mentioning that it may be a kernel bug.

There are a couple of solutions I’ve seen for this so far.

Use the nomodeset option in the boot options to force the system to use VESA drivers  – more info at – time is of the essence for me so I didn’t try this but it looks like it may work.  You will still need to fix the Nvidia driver after you have completed the install.

My system has an additional Nvidia card so it can be used as a multi display system. As I’m building this system as a headless Splunk server I don’t need multi displays on this system so I just removed the card and used the default (on board) VGA display.

For me, this solved the problem and I was immediately able to boot the X64 live CD and start the install.

I have another box with the same config that I will need to configure multi display.  I have more time to build that box so I will dig a bit deeper into the cause of the issue and hopefully be albe to resolve the issue.  I’ll post what I discover.

As alway comments and questions are welcome.

Categories: General Tags: , , , , ,

Firefox 4 Tips

March 19, 2011 Comments off

Firefox 4 ImageThe final version of Firefox 4.0 is scheduled for release on March 22nd so I upgraded to the latest Firefox 4.0 Release Candidate last night and so far so good.  I like the new look and the speed of the new version.  It also has some great new features such as:

  • New Tab Location
  • Tabs for applications
  • Easy switch to tab
  • Synchronizing Settings
  • Managing Add-ons made easy
  • Panorama
  • HTML 5 support
  • Full Hardware Acceleration
  • Organizing tabs made easy
  • Firefox button for easy access and reduced clutter.

Extensions usually take a little while to be updated but I was able to install all the ones that I use regularly including Delicious.

I’ve also outlined below a couple of tips I used to get everything working.

1) Tab width

It used to be possible to change the Tab width in the about:config setup however the current Firefox 4 release candidate appears to now ignore this.  Fortunately someone has created an add-on that will fix this for you and allow you to update the tab widths.  Here’s the link

2) Delicious Bookmark Add-on

The Delicious bookmark add-on works with Firefox 4.0b3pre but currently doesn’t install on Firefox 4.0 RC however it is possible to force it to install and so far it’s working fine for me.


First goto the addon site and choose install anyway.

The browser will complain that it can’t be installed.

Within FireFox 4 goto “Help” in the Menu, then “Troubleshooting Information”

Click on the button “Open Containing Folder” (within the box “Application Basics” on top of the page)

Close Firefox

Goto into directory “extensions\{2fa4ed95-0317-4c6a-a74c-5f3e3912c1f9}”

Within the file install.rdf change

em:maxVersion=”4.0b3pre” />


em:maxVersion=”4.0″ />

Restart Firefox

The above worked perfectly for me and once Firefox was restarted the extension was enabled.  I then manually added the tag and bookmark buttons to the toolbar.

Thanks to the users on the delicious support forum

I’m always looking to improve Firefox so please share your tips or favourite add-ons in the comments section below.

Categories: Technology Tags: , , ,