Building Python3 modules from FreeBSD ports

If you're like me, you will have python2.7 selected as the default interpreter in your TrueOS installation. However, some programs may require Python version 3 (and associated python3 modules). However, there aren't many py34 modules available using pkg install.

The solution to the problem is simple if you have a FreeBSD ports tree installed.

To build a port for a specific version of python, execute:

PYTHON_VERSION=pythonX.X make

Substitute X.X for the appropriate version that you require. Instead of the default py27-modulename, FreeBSD will build pyXX-modulename.

The default, of course, can be changed in make.conf (see man make.conf for details.)

Extracting structured data (in a table) from HTML5 using BeautifulSoup / Python

I recently ripped a CD that was unknown to my CDDB server. I found a web page that contained a track list, but found it very cumbersome to copy and paste the information due to the formatting of the web page.

Consequently, I opened up the page using the Firefox DOM inspector, and noticed that each title was associated with HTML class 'title'. Surely, the data element of interest could be extracted using some higher-level language!

I elected to do some research and discovered that I could solve this problem, easily, using Python 2.7 and BeautifulSoup.

After some research (having never used BeautifulSoup before), this is the unbelievably simple script that I came up with:

from requests import get
from bs4 import BeautifulSoup
url = 'https://rainforroots.bandcamp.com/album/the-kingdom-of-heaven-is-like-this'
htmlString = get(url).text
html = BeautifulSoup(htmlString, 'html5lib')
tags = html.find_all('div', {'class':'title'})
text = [t.get_text() for t in tags]
print str(len(text)) + ' items matched:\n'
# join(j.split()) is a quick hack to remove excess whitespace
for i,j in enumerate(text): print ' '.join(j.split())

WOW! Clearly, this is a useful library.

pyWebDav <-> Windows 8.1: one-line file sharing, after a couple of registry tweaks

How often do you want to share a directory of files on one (*nix) system, and mount it as a network drive on a Windows host, with minimum hassle? For me, this is a common occurrence. After a couple of adjustments to my Windows 8 machine, I can do this after installing the pywebdav module, making for a one-liner that creates a temporary high-throughput network share that can be mounted as a drive in Windows.

The annoying part of this trick is that it takes a couple of registry tweaks, and this may affect other resources shared through WebClient, notably, SharePoint.

The easy part of this is installing the pyWebDav Python package, and running a one-line command to start a WebDAV server. Install this (in BSD we type pkg install py27-PyWebDAV). The -D option specifies the directory to share, and I like to know what the program is thinking (verbose with -v option):

davserver -D directory_to_share -n -H server_ip_address_here -v

davserver needs the IP address of the network resource that it will use to listen for connection requests. You can test the server using the loopback address and a webdav client installed on the same machine.

Now for the more annoying part: registry changes, with some commentary. On a Windows machine, navigate to the following subkey in regedit:

HKLM/System/CurrentControlSet/Services/WebClient/Parameters

Modify these values:

SupportLocking=0x0 (default=0x1) 

REQUIRED for interoperability with pyWebDav. 

• davserver's -J parameter appears buggy / nonfunctional with Windows 8, so we have to modify the registry.

FileSizeLimitInBytes=0xffffffff (or whatever, default is 50 000 000d)

Optional, but you should change this.

• The default value of this is only 50 megabytes, which is really useless for ISOs or whatnot. I recommend 4.2 gigabytes, which is what this value represents. A bigger value would be preferable. Fix this, Microsoft! We've moved on to use terabyte devices.

BasicAuthLevel=0x2

Optional change, useful if you add basic authentication (which pyWebDav supports)

• 0 - Basic authentication disabled
• 1 - Basic authentication enabled for Secure Sockets Layer (SSL) shares only (default)
• 2 or greater - Basic authentication enabled for SSL shares and for non-SSL shares

Now that you've made these changes, you can restart the WebClient service. Launch cmd with administrator privileges and type:

net stop  WebClient
net start WebClient

 Mount the share (note that you'll need to do this in a cmd shell WITHOUT administrator privileges for your user to see the drive):

net use * http://server_ip_address:8008/

You can also map network drives using a more GUI method.

Sweet!

Dynamic DNS update Python script, for use with a home gateway device

I use the free ZoneEdit service to keep a dynamic DNS entry up-to-date. I also now use a Netgear WNDR4000 wireless router to connect to the Internet through my cable modem.

I have authored a script that uses HTTP basic authentication to grab the IP address from my router, compare it to the last polled IP address, and update ZoneEdit with the new IP if it has changed. I have this script set to run every minute.

#!/usr/local/bin/python

import urllib2, re, tempfile, os

# User variables - change these to fit your router
debug=False

router_user=""
router_passwd=""
router_webpage="http://192.168.0.254/RST_st_dhcp.htm"

zoneedit_user=""
zoneedit_passwd=''
zoneedit_host=""

# Returns a string containing webpage contents
def fetch_webpage(location, username, password):
  passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
  passman.add_password(None, location, username, password)
  authhandler = urllib2.HTTPBasicAuthHandler(passman)
  opener = urllib2.build_opener(authhandler)
  urllib2.install_opener(opener)
  return urllib2.urlopen(location).read()


s=fetch_webpage(router_webpage, router_user, router_passwd)
pattern = re.compile("IP Address.*?([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})", re.DOTALL)
m = pattern.search(s)
newip = m.group(1)
if debug: print "Current IP Address: " + newip

filename = tempfile.gettempdir() + "/router_ip"
flags = "r+"
if not os.path.exists(filename):
  flags = "w+"
f = open(filename, flags)

oldip = f.read().strip()

if (cmp(oldip,newip) == 0):
  if debug: print "Old IP is the same (" + oldip + ")"

else:
  web_result = fetch_webpage('https://dynamic.zoneedit.com/auth/dynamic.html?host='+zoneedit_host, zoneedit_user, zoneedit_passwd)
  if debug: print web_result

  f.seek(0)
  f.write(newip)