Archive

Archive for the ‘Technology’ Category

Building Useful Agents 01

December 12, 2016 Leave a comment

This is a POC. Hunting through multiple pages of content while sites throw up multiple random pages triggered by javascript is very annoying. This useful agent requires a file called show_list.txt including PERL regular expressions if you are so inclined. It’s a basic list of shows you want it to find.

eg.g1gz0
3 Percent
^Aftermath S0
Agent X
Aquarius (US)*
Ash vs Evil Dead
Better Call Saul
Blindspot S0
DC[s]* Legends
Elementary S0

The file is read into the script and all the spaces are replaced by \W* to cope with . _ etc.

This script will load and parse the website looking for words matching the expressions and create a HTML page with links.

The last part of the script also looks for the latest version of the file I currently have and then I can decide whether I need a new version or not.

Now I can run this agent to the page depth I define and it will return matches based on the shows in show_list.txt

At a page depth of 1 it runs in about 5 seconds, at a page depth of 4 it runs in about 10 seconds. Which is so much more efficient than manually checking those pages myself.

This can be be extended (and you can also use other tools that have been around for a long time) however I find that I usually learn more building it myself.

#!/bin/bash
#useful sites:
# Multi line grep http://stackoverflow.com/questions/2686147/how-to-find-patterns-across-multiple-lines-using-grep
# command: pcregrep -M '<tr name="hover" class="forum_header_border">.*(\n|.)*?</tr>' homepage.txt
# Helpful site for regex https://regex101.com/

if [ $# -eq 0 ]; then
echo "No arguements supplied - defaulting for 4 pages"
PAGES=4
else
PAGES=$1
fi

echo $PAGES

SHOW_LIST=""
COUNT=0
while IFS='' read -r line || [[ -n "$line" ]]; do
echo -n "Text read from file: $line"
SHOW_LIST1=$(echo $line | sed 's_[.\ ]_\\W*_g')
echo " --> $SHOW_LIST1"
if [ $COUNT -eq 0 ]; then
SHOW_LIST=$SHOW_LIST1
else
SHOW_LIST="$SHOW_LIST|$SHOW_LIST1"
# echo $SHOW_LIST

fi
((COUNT ))
done < "show_list.txt"
echo $SHOW_LIST

SHOWS=$SHOW_LIST
URL="https://aaaaa.ag/page_&quot;
RESULT=""
for i in $(seq 0 $PAGES);
do
if [ $i -eq 0 ]; then
RESPONSE=$(curl -s "https://aaaaa.ag/&quot;)
RESULT=$(echo "$RESPONSE" | grep -iP "$SHOWS" | grep -iP "download|magnet" )
echo result="$RESULT"
echo -------------
else

URL1=$URL$i
echo $URL1
PAGE="page_$i.txt"
echo $PAGE
# RESPONSE=$(cat $PAGE)
RESPONSE=$(curl -s $URL1)
#RESULT1=$(echo "$RESPONSE" | grep -iP "$SHOWS" | grep -i download | grep -v 720p)
RESULT1=$(echo "$RESPONSE" | grep -iP "$SHOWS" | grep -iP "download|magnet")
echo result1="$RESULT1"
RESULT="$RESULT$RESULT1"
echo \n result="$RESULT"
echo -------------
fi
done

echo
echo ****** FINAL ******

echo "$RESULT"
echo "$RESULT" > result.txt

OUTPUT=$(echo "$RESULT" | sed s/title=\"/\>/g | sed s/\"\>\</\</g | sed s/a\>/a\>\<p\>/g)
echo
echo
echo "$(date) <BR>PAGES=$PAGES<P>" > _show_list.html
echo $OUTPUT >> _show_list.html

---- you will need to customise this for your system -----
echo "<P><P> File List <P><P>" >> _show_list.html
echo "<table border=0>" >> _show_list.html

FILES=$(find /data01/download/complete/ -maxdepth 5 -printf "%f\n")
#echo "$FILES"

for f in $(echo $SHOW_LIST | sed 's/|/ /g')
do
NICE_FILE=$(echo $f | sed 's/\\W\*/ /g')
echo "<TR><TD>" >> _show_list.html
echo -n "$NICE_FILE</TD>" >> _show_list.html
FILE_NAME=$(echo "$FILES" | sed 's/[.-]/ /g' | grep -iP "$f" | sort -rf | head -1)
echo "<TD> $FILE_NAME </TD></TR>" >> _show_list.html

done
echo "</table>" >> _show_list.html
#echo "<P><P><P>ALL FIELS<P><P>$FILES" >> _show_list.html

cp _show_list.html /data01/download/complete/_showlist.html

echo "**** COMPLETE ****"

Categories: Technology Tags: , ,

Stop almost all YouTube and internet ads in less than 30 minutes using a Squid Proxy Server

October 17, 2015 Leave a comment

no andsI use Adblock or Adblock Plus in all my web browsers so I hardly ever see an ad but when I’m using the web however when I (or the family) use Apple, Android or XBox devices to watch youtube content online or browse the internet I’m constantly reminded how annoying ads actually are.

Someone in my family (not me) was trying to watch Channel 10 catchup TV today day and before they could start watching the show they had to sit through 2 ads and then there were more ads (up to 4-5 ads at a time) throughout the broadcast as well.

That is a rubbish experience & I knew I could make it better.

I’d been putting it off for a long while – mostly due to time constraints – but when I actually focused on enabling ad blocking via my Squid proxy it took less than 30 minutes to block approx 95% of all ads supported by adblock plus including youtube, google, etc and for right now – my family is happy that they can watch Channel 10 catchup TV without any ads & with no other counter measures in place approx 95% of all web based ads (including youtube ads on apple devices etc)  have been blocked on local apple devices using the proxy.

If (like me) you already have a Squid server running – I highly recommend – https://calomel.org/squid_adservers.html – just make sure you put their recommendations at the top of your squid.conf file – otherwise it didn’t work for me.  I also had to add a few additional domains to the ad_block.txt file to block some of the local advertising.

The main reason I’m excited about this is that I can apply a single rule to ignore a new ad server and have it apply across all the devices on my local network that didn’t have any adblocking capability previously.

I’m also aware that there are many ways to circumvent this this type of blocking – however – until then – everyone on my lan is enjoying ad free browsing.

I’m thinking about proving a Raspberry Pi turnkey solution as a side project, would you be interested?

Categories: General, Technology Tags: ,

Hard Disk Drive Data Recovery Using A Freezer Actually Works

February 4, 2015 1 comment

Frozen HDDAfter a Linux kernel upgrade, an old data hard drive in one of my Linux boxes (which I very rarely reboot) failed last night so the box  wouldn’t boot without manually skipping the HDD.  Recovering the Linux box simply meant removing the old data drive, swapping it with another drive I had lying around, updating the drive UUID in /etc/fstab and away the server went again.  Plus the server got a 320GB data drive upgrade in the process 🙂

Loosing the HDD data drive wasn’t a really big issue as the drive was automatically backed up about a week ago and it only stores media files and other transient data so it wasn’t a huge loss if I couldn’t recover it however, it would be much easier if I could get the last week of data off the drive.

Plugging the problem drive into an external HDD case and trying to mount it on another machine was no good.  I could hear the drive spinning up and the hdd heads trying to engage with the disk platters but it just wouldn’t start.  Then, I remembered a very old (but fairly well known trick) which is to put the HDD in the Freezer for an hour or so and then retry accessing the drive.  I’d never had the opportunity (or reason) to try this before and I though it was something worth testing so I put the HDD in a sealed zip lock plastic bag and stuck it in the freezer.

After an hour I removed the HDD from the zip lock, plugged it back into the external HDD case and connected it to my PC.  I fired up a System Rescue Linux VM and attempted to attach the HDD.  I could hear the drive spin up and the heads were now engaging with the disk platters, VMWare reported that the drive was attached and available but the Linux VM wouldn’t read the drive (fdisk -l just hung).  After about 5 minutes and a few restarts the drive it was finally recognised by Linux and I was able to copy the data off the drive.

I must admit, I’d heard the rumour that freezing a hard drive would allow it to work for long enough to copy the data off it many times before but now that I’ve actually tested it myself I can confirm that it does actually work.

Final thoughts, even though I had a backup it was still a week old, make sure you backup your data regularly as you never know when a drive might fail and also ensure you backup any important data BEFORE a server restart.

Categories: Technology Tags: , , , ,

Xbox Live Status Script

December 27, 2014 Leave a comment

If you’re like me, frustrated by all the Xbox One DDOS going on over the Christmas Holidays and have a linux box lying around here is a small script to check the service status in Australia.

Just cron it and output it to a log file – after that you can do what ever you want with the output.

I hooked the output up to my Raspberry Pi and changed the color of an LED  from Red to Green based on the Xbox Live Core Services Status.

#!/bin/bash
# Author: Merric Reese
# Date: Dec 26, 2014
# Purpose: Show Xbox Live Status and report when Core Services are online
# Notes: Common sense rules apply to usage - I only run this script about every 5 minutes during a service outage 
MR_PAGE=$(curl -s http://support.xbox.com/en-AU/xbox-live-status)
MR_XBOX_CORE=$(echo "$MR_PAGE" |grep -Eo "Xbox Live Core Services.*[L|l]imited|Xbox Live Core Services.*[A|a]ctive" |grep -iEo "active|limited")
MR_XBOX_PURCHASES=$(echo "$MR_PAGE" |grep -Eo "Purchase and Content Usage.*[L|l]imited|Purchase and Content Usage.*[A|a]ctive" |grep -iEo "active|limited")
MR_XBOX_GAMING=$(echo "$MR_PAGE" |grep -Eo "Social and Gaming.*[L|l]imited|Social and Gaming.*[A|a]ctive" |grep -iEo "active|limited")
MR_XBOX_WEBSITE=$(echo "$MR_PAGE" |grep -Eo "<h3>Website.*[L|l]imited.*</h3>|<h3>Website.*[A|a]ctive.*</h3>" |grep -iEo "active|limited")
MR_XBOX_TV=$(echo "$MR_PAGE" |grep -Eo "TV, Music and Video.*[L|l]imited|TV, Music and Video.*[A|a]ctive" |grep -iEo "active|limited")
if [ "$MR_XBOX_CORE" != "Limited" ] && [ "$MR_XBOX_GAMING" != "Limited" ]; then
# echo "#"
echo "### FIXED ### " $(date) "Xbox Live Core Services: $MR_XBOX_CORE | Social and Gaming: $MR_XBOX_GAMING | Purchases and Content: $MR_XBOX_PURCHASES | Website: $MR_XBOX_WEBSITE | TV, Music and Video: $MR_XBOX_WEBSITE"
# echo "#"
else
echo "waiting... " $(date) "Xbox Live Core Services: $MR_XBOX_CORE | Social and Gaming: $MR_XBOX_GAMING | Purchases and Content: $MR_XBOX_PURCHASES | Website: $MR_XBOX_WEBSITE | TV, Music and Video: $MR_XBOX_WEBSITE"
fi

BTW: I haven’t spent the whole time coding or waiting for Xbox Live to come back online.  I’d just like to able to download the free games when the service is available and this was an easier way to check the service status.

If you found this useful then please leave a comment.

Have a very Merry Christmas and a Happy New Year.

Categories: Technology Tags: , , ,

Panasonic PT-AE900E fan noise is actually the Dynamic Iris

July 5, 2014 4 comments

Panasonic

My Panasonic PT-AE900E projector (same as PT-AE900U) is getting a bit long in the tooth and started making a noise that sounded like a PC power supply fan that is about to die (or a couple of marbles rolling around the bottom of the projector).  The sound appears as soon as the projector is turned on and remains there until the project is switched off.

I pulled the projector apart (not for the faint-heared) to see if I could find the fan that was about to cease up but they all seemed fine.

After some additional research on Google I found this AVS forum discussing a similar noise and it suggested disabling the dynamic iris as that could be the cause of the issue.  I eventually found the PT-AE900E Service Manual (thanks to Garry Tan’s blog post).

Fortunately, I also found this Youtube Video on a fix for the dynamic iris an older model Panasonic projector.  The iris control is a little bit harder to get to on the AE900E but the iris control is accessed through the lamp cover the same way.

The video suggests using CRC or WD40 on a cotton bud but I chose to use some sewing machine oil.

When I first moved the iris it took about 1/2 a second to go from fully open to fully closed.  After lubricating the iris shaft and spring (as per the video) it now shuts in a couple of milliseconds.

After closing up the projector and testing it the noise is completely gone & the dynamic iris is now working as expected.

I also found this Spare Parts suppler in Australia in case you need a replacement iris.

Usual warnings – Make sure the projector is unplugged before performing any of the above, be extremely careful and don’t try this unless your confident you know what you are doing.

 

Firefox 4 Tips

March 19, 2011 Comments off

Firefox 4 ImageThe final version of Firefox 4.0 is scheduled for release on March 22nd so I upgraded to the latest Firefox 4.0 Release Candidate last night and so far so good.  I like the new look and the speed of the new version.  It also has some great new features such as:

  • New Tab Location
  • Tabs for applications
  • Easy switch to tab
  • Synchronizing Settings
  • Managing Add-ons made easy
  • Panorama
  • HTML 5 support
  • Full Hardware Acceleration
  • Organizing tabs made easy
  • Firefox button for easy access and reduced clutter.

Extensions usually take a little while to be updated but I was able to install all the ones that I use regularly including Delicious.

I’ve also outlined below a couple of tips I used to get everything working.

1) Tab width

It used to be possible to change the Tab width in the about:config setup however the current Firefox 4 release candidate appears to now ignore this.  Fortunately someone has created an add-on that will fix this for you and allow you to update the tab widths.  Here’s the link https://addons.mozilla.org/en-us/firefox/addon/custom-tab-width/.

2) Delicious Bookmark Add-on

The Delicious bookmark add-on works with Firefox 4.0b3pre but currently doesn’t install on Firefox 4.0 RC however it is possible to force it to install and so far it’s working fine for me.

SOLUTION:

First goto the addon site and choose install anyway.

https://addons.mozilla.org/en-US/firefox/addon/delicious-bookmarks/

The browser will complain that it can’t be installed.

Within FireFox 4 goto “Help” in the Menu, then “Troubleshooting Information”

Click on the button “Open Containing Folder” (within the box “Application Basics” on top of the page)

Close Firefox

Goto into directory “extensions\{2fa4ed95-0317-4c6a-a74c-5f3e3912c1f9}”

Within the file install.rdf change

em:maxVersion=”4.0b3pre” />

to

em:maxVersion=”4.0″ />

Restart Firefox

The above worked perfectly for me and once Firefox was restarted the extension was enabled.  I then manually added the tag and bookmark buttons to the toolbar.

Thanks to the users on the delicious support forum http://support.delicious.com/forum/comments.php?DiscussionID=5168

I’m always looking to improve Firefox so please share your tips or favourite add-ons in the comments section below.

Categories: Technology Tags: , , ,

Wireless-N Upgrade – Thinking about a Christmas Treat

December 23, 2010 Comments off

Why Not?

Lately I’ve been looking for a bit more bandwidth in my home network to be able to stream HD video over wireless and moving to N seemed like the logical choice.

For the last 3 years I’ve run a Linksys WRT54GL with DD-WRT firmware and it has been solid as a rock – best wireless I’ve ever had and will be a very hard act to follow.

My “new” work laptop supports Wireless-N and I get about 150MB/s at the office (which is faster than my wired lan at home) so I thought it could be a potential solution to my bandwidth problem.

So needing a bit more wireless bandwidth – the search was on for a new DD-WRT compatible wireless solution.

I’m currently considering the ASUS RT-N16 – seeems to have excellent specs – plenty of grunt, 2 USB ports and supports either DD-WRT or Tomato firmware.  From what I’ve read the ASUS firmware is flaky and results in wireless dropouts but those on Whirlpool that have upgraded to either Tomato or DD-WRT are reporting good performance.  In addition the RT-N16 has a fast processor (480MHz stock + ability to overclock), 32 MB Flash (which is huge – even the biggest installs of DD-WRT are only just over 8MB) – to me this seems like a really fast piece of kit capable of running DD-WRT.  The 2 USB ports bring into play usage as a torrent box plus a printer hub.

The ASUS RT-N16 seems to have a hugh amount of potential – I think I’m going to have to get one and see how far I can stretch it.

Anyone else have experience with this model? Other thoughts?