Wednesday, November 18, 2015

Microsoft Windows 10 (build 1511 / 10586, November 2015) update likely fails if you "forcefully removed" applications in Windows 10

Months after I had followed directions posted on Super User to remove arguably useless parts of Windows 10 (including Cortana), it seemed that Windows Update didn't work very well on this particular Windows 10 update. Executing the Get-WindowsUpdateLog command in PowerShell rendered a log file on my desktop that didn't contain anything useful.

I suppose that the answer to this problem is to use the Windows Media Creation Tool to perform the update, but without diagnostic logs to demonstrate where the failure was, I am reluctant to spend time attempting this.

Saturday, October 24, 2015

One minute tutorial: Getting started with a fresh NodeMCU ESP8266 (direct from eBay)

The NodeMCU / ESP8266 is a ridiculously cheap ($2) micro-controller that can talk to an 802.11N wireless network. This device is the herald of the "Internet of Things."

If you buy one on eBay, chances are it will NOT be flashed with NodeMCU firmware (it speaks Lua). This understandably confused the crap out of me. So, this serves as a guide to those running Manjaro (Arch) Linux:

  1. Download NodeMCU firmware
    "integer" versions will NOT understand floating point arithmetic, but save on system resources.
  2. Install minicom
    pacman -Ss minicom
  3. Install esptool from the AUR:
    yaourt esptool-git
  4. Make sure your Linux box recognizes the device
    tail -f /var/log/messages.log
  5. Plug in the NodeMCU device. You should see:
    kernel: usb whatever: ch341-uart converter now attached to ttyUSBX

    ch341-uart converter detected

    If this didn't happen, you cannot proceed and must troubleshoot.
  6. Flash the firmware. Make sure you have permission to read and write to the device in the command below. When in doubt, run as root:
    esptool.py --port /dev/ttyUSBX write_flash 0x00000 nodemcu_whatever.bin
    where:
    X corresponds to the device from the log file above, and
    nodemcu_whatever.bin corresponds to the firmware file in step 1.
  7. Unplug the USB cable after the flash command from the previous step is completed, and plug it back in.
  8. Fire up minicom, and talk to the board at 9600 baud:
    minicom -D /dev/ttyUSBX -b 9600
  9. Press the "enter" key a few times. In response, you should see:
    >
    >
  10. Try "hello world":
    > print ("hello world")
    hello world
    >

  11. Now you can configure wireless:
    > wifi.setmode(wifi.STATION)

    > wifi.sta.config("SSID", "PASSWORD")

    > ip, nm, gw=wifi.sta.getip()

    > print (ip, nm, gw)
    this   is   amazing
  12. Do some interesting things!

Wednesday, October 7, 2015

How to tell the difference between crocodiles and alligators

Such things are important when you have young children, so I authored a rhyme (with help of a friend on Facebook):
Crocs always show both rows of teeth
And swim in salty sea.
When gators smile, you see a "U,"
With crocs, you see a "V"

With dotted snout in muddy swamp,
the Gator lies happily.
Crocs, with spots from head to toe,
Live life... estuarily.

Friday, October 2, 2015

Extracting structured data (in a table) from HTML5 using BeautifulSoup / Python

I recently ripped a CD that was unknown to my CDDB server. I found a web page that contained a track list, but found it very cumbersome to copy and paste the information due to the formatting of the web page.

Consequently, I opened up the page using the Firefox DOM inspector, and noticed that each title was associated with HTML class 'title'. Surely, the data element of interest could be extracted using some higher-level language!

I elected to do some research and discovered that I could solve this problem, easily, using Python 2.7 and BeautifulSoup.

After some research (having never used BeautifulSoup before), this is the unbelievably simple script that I came up with:

from requests import get
from bs4 import BeautifulSoup
url = 'https://rainforroots.bandcamp.com/album/the-kingdom-of-heaven-is-like-this'
htmlString = get(url).text
html = BeautifulSoup(htmlString, 'html5lib')
tags = html.find_all('div', {'class':'title'})
text = [t.get_text() for t in tags]
print str(len(text)) + ' items matched:\n'
# join(j.split()) is a quick hack to remove excess whitespace
for i,j in enumerate(text): print ' '.join(j.split())

WOW! Clearly, this is a useful library.

Saturday, September 26, 2015

Making old machines run a modern Linux distribution surprisingly well (using Zswap)

In between study sessions, I've been experimenting with some old hardware that I want to get rid of on eBay. I happen to have managed to get an old IBM A31 with 384 MB of RAM up and browsing the Web while using a modern operating system, namely: Manjaro Linux 0.8.13 (LXDE), kernel version 4.2.

The challenges here are plenty: low RAM, extremely slow and old rotational hard drive (20 GB), and the ever-increasing presence of complex multimedia and Javascript on the modern Web.

The key to my success has been the use of Zswap. In a nutshell, Zswap compresses pages that would be normally swapped out of RAM (and onto disk) into a separate storage area (in RAM) that is first compressed before being queued for swapping. This effectively increases the RAM of the computer in exchange for CPU time. With a system that is heavily I/O constrained, this is a very reasonable trade-off to make.

In order to make the default installation of Manjaro XFCE functional under such constraints:
  1. Edit /etc/default/grub and configure zswap to use the more space-efficient deflate compressor:
    1. Append "zswap.enabled=1 zswap.compressor=deflate zswap.max_pool_percent=90" to GRUB_CMDLINE_LINUX_DEFAULT
    2. update-grub
  2. Add the deflate module to the initial ramdisk by editing /etc/mkinitcpio.conf:
    1. Append "deflate" to MODULES
    2. Look inside of /etc/mkinitcpio.d/ to determine what profile to specify for rebuilding the ramdisk. If you are using the Linux 4.2 kernel (recommended):
      mkinitcpio -p linux42
  3. Uninstall the resource-hungry pulseaudio subsystem:
    1. pacman -R pulseaudio pulseaudio-ctl pulseaudio-alsa
  4. Install and use QupZilla as the default web browser
    1. pacman -Sy qupzilla
  5. For even better performance, change QupZilla's User-Agent string to an Android Webkit-based mobile device:
    1. Edit -> Preferences -> Other -> Change browser identification / User Agent Manager -> Change global User Agent

      I suggest:
      Mozilla/5.0 (Linux; U; Android 4.0.3; en-us; HTC Sensation Build/IML74K) AppleWebKit/538.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/538.1
Remarkably, these changes allow a full Web experience. Even Youtube videos seem to function.

On such a machine -- and in order to reduce slow disk I/O -- it might be advantageous to use a btrfs root filesystem mounted with gzip compression.

Wednesday, September 16, 2015

ZFS compression results in workload starvation, partially ameliorated by async_write_max_active

I have a particular dataset that consists of old backup files. Rather than storing them in .tar.gz archives, where the contents remain buried (and the files cannot be culled with useful tools such as fdupes), I elected to create a zfs dataset that uses gzip-9 compression.

However, on my system, I noticed that writing to the compression=gzip-9 zfs dataset resulted in starvation to other I/O processes  -- in particular, read processes -- rendering the system unusable.

In order to analyze the situation, I used this dtrace script to analyze different classes of I/O operations on my system.

See below for the sysctl tuneable that I modified.

vfs.zfs.vdev.async_write_max_active=10 vfs.zfs.vdev.async_write_max_active=3
  Delete                                            
           value  ------------- Distribution ------------- count    
             256 |                                         0        
             512 |@@@                                      2        
            1024 |@@@@                                     3        
            2048 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@        24       
            4096 |                                         0        

  Flush                                             
           value  ------------- Distribution ------------- count    
             512 |                                         0        
            1024 |@@@@                                     4        
            2048 |@                                        1        
            4096 |@@@@                                     4        
            8192 |@@@@                                     4        
           16384 |@@@                                      3        
           32768 |@@@@@@@@@@@@@@@                          14       
           65536 |@@@@@@@@                                 7        
          131072 |                                         0        

  Write                                             
           value  ------------- Distribution ------------- count    
              16 |                                         0        
              32 |                                         96       
              64 |@                                        531      
             128 |@@                                       735      
             256 |@@                                       1067     
             512 |@@@@@@@@@@@@@@@@@@@@@@@@@@               11639    
            1024 |@@@                                      1573     
            2048 |@@                                       878      
            4096 |@@                                       785      
            8192 |@                                        612      
           16384 |                                         183      
           32768 |                                         25       
           65536 |                                         16       
          131072 |                                         13       
          262144 |                                         3        
          524288 |                                         0        

  Read                                              
           value  ------------- Distribution ------------- count    
              32 |                                         0        
              64 |                                         12       
             128 |                                         48       
             256 |@                                        124      
             512 |@@@@@@@@@@                               2046     
            1024 |@@@@                                     906      
            2048 |@@@@@@                                   1244     
            4096 |@@@@@@@                                  1523     
            8192 |@@@@@@                                   1228     
           16384 |@@@@                                     774      
           32768 |@                                        173      
           65536 |                                         26       
          131072 |                                         22       
          262144 |                                         5        
          524288 |                                         0
  Delete                                            
           value  ------------- Distribution ------------- count    
             256 |                                         0        
             512 |@@                                       1        
            1024 |@@@@@@@@@@@@@@@@@@@@                     12       
            2048 |@@@@@@@@                                 5        
            4096 |@@                                       1        
            8192 |@@@@@@@@                                 5        
           16384 |                                         0        

  Flush                                             
           value  ------------- Distribution ------------- count    
             512 |                                         0        
            1024 |@@@@@@@@@                                5        
            2048 |@@@@                                     2        
            4096 |@@@@@                                    3        
            8192 |@@@@@@@@@                                5        
           16384 |                                         0        
           32768 |@@@@@@@                                  4        
           65536 |@@@@@                                    3        
          131072 |                                         0        

  Write                                             
           value  ------------- Distribution ------------- count    
              16 |                                         0        
              32 |@                                        468      
              64 |@@@@@@                                   3741     
             128 |@@@@@@                                   3627     
             256 |@@@@@                                    3234     
             512 |@@@@@@@@@@@@@@@@@                        10520    
            1024 |@                                        916      
            2048 |@                                        608      
            4096 |@                                        476      
            8192 |@                                        476      
           16384 |                                         246      
           32768 |                                         80       
           65536 |                                         54       
          131072 |                                         11       
          262144 |                                         5        
          524288 |                                         0        

  Read                                              
           value  ------------- Distribution ------------- count    
              32 |                                         0        
              64 |                                         18       
             128 |@                                        70       
             256 |@                                        153      
             512 |@@@@@@@@                                 890      
            1024 |@@@@                                     445      
            2048 |@@@@@                                    594      
            4096 |@@@@@@                                   667      
            8192 |@@@@@@                                   709      
           16384 |@@@@@                                    609      
           32768 |@@@                                      381      
           65536 |@                                        130      
          131072 |                                         26       
          262144 |                                         10       
          524288 |                                         0
 
                              avg latency      stddev        iops  throughput
Write                               1892us      8057us       302/s    33416k/s
Delete                              2303us       732us         0/s       34k/s
Read                                7747us     15904us       135/s    17011k/s
Flush                              42742us     37072us         0/s        0k/s

                               avg latency      stddev        iops  throughput
 Write                               1521us      8316us       407/s    30107k/s
 Delete                              3557us      3001us         0/s       41k/s
 Read                               14255us     27405us        78/s     8844k/s
 Flush                              25081us     32016us         0/s        0k/s
In particular, it seems that the average latency of I/O Flush dropped considerably.

A more exhaustive study under more carefully controlled conditions seems like a reasonable next step.

Wednesday, August 12, 2015

dtrace newbie HOWTO: listing providers AND structures

One of the first things that I wanted to learn about dtrace was the answer to the question: "How do I figure out where I can put hooks to trace system activities?"

This is accomplished with 'dtrace -l', which lists all the possible providers to which dtrace scripts can be attached.

Once you find a provider of interest, e.g. io:::start(), you can again use the list (-l) option to take a peek into the kind of data that is accessible to you:

# dtrace -lvn io:::start

   ID   PROVIDER            MODULE                          FUNCTION NAME
62099         io            kernel                                   start

    Probe Description Attributes
        Identifier Names: Private
        Data Semantics:   Private
        Dependency Class: Unknown

    Argument Attributes
        Identifier Names: Private
        Data Semantics:   Private
        Dependency Class: ISA

    Argument Types
        args[0]: struct bio *
        args[1]: struct devstat *


The next logical question is, "What is in struct bio? What is in struct devstat?"

You could find the answer by grepping through /usr/include/sys. However, there's a much faster way to find out what you can look at. This is the critical piece of information that I couldn't find in other HOWTO manuals on the Internet:

# dtrace -qn 'io:::start{print(*args[0]); exit(0); }'

struct bio {
    uint8_t bio_cmd = 0x2
    uint8_t bio_flags = 0
    uint8_t bio_cflags = 0
    uint8_t bio_pflags = 0
    struct cdev *bio_dev = 0
    struct disk *bio_disk = 0xfffff80012144000
    off_t bio_offset = 0x116948cb000
    long bio_bcount = 0x1000
    caddr_t bio_data = 0xfffffe0005ada000
    struct vm_page **bio_ma = 0
    int bio_ma_offset = 0
    int bio_ma_n = 0
    int bio_error = 0
    long bio_resid = 0
    void (*)() bio_done = kernel`g_disk_done
    void *bio_driver1 = 0
    void *bio_driver2 = 0
    void *bio_caller1 = 0
    void *bio_caller2 = 0
    struct bio_queue = {
        struct bio *tqe_next = 0
        struct bio **tqe_prev = 0
    }
    const char *bio_attribute = 0
    struct g_consumer *bio_from = 0
    struct g_provider *bio_to = 0
    off_t bio_length = 0x1000
    off_t bio_completed = 0
    u_int bio_children = 0
    u_int bio_inbed = 0
    struct bio *bio_parent = 0xfffff8018178f2e8
    struct bintime bio_t0 = {
        time_t sec = 0x24d28
        uint64_t frac = 0x4543b859cc1368f0
    }
    bio_task_t *bio_task = 0
    void *bio_task_arg = 0
    void *bio_classifier1 = 0
    void *bio_classifier2 = 0
    daddr_t bio_pblkno = 0x8b4a4658
}


Now, THAT was easy!

Tuesday, August 11, 2015

Fixing a rogue Windows 7 Out of Box Experience (OOBE)

I just finished reinstalling Windows 7 OEM on a PC. The result was that the installation failed to work correctly, leaving me in a "reboot loop," because my hardware vendor included the wrong device driver for my keyboard on the installation medium.

The solution?

  1. Press SHIFT-F10 after the error box to open up a "cmd" prompt.
  2. Execute regedit.exe
  3. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\Setup
  4. Delete value for CmdLine
  5. Change value to zero for OOBEInProgress, RestartSetup, SetupPhase, and SetupType.

I later had to enable the Administrator account (see previous blog post).

Windows 7: Administrator account is disabled. What now?

I recently had some problems with a Windows 7 installation gone bad that required some trickery that left the system in a place where no user accounts were created, and the Administrator account was disabled. I could, therefore, only restart the computer.

The trick to solving the problem was to hit F10 to bring up the NTLDR advanced boot options prompt (for some reason, F8 to start Safe Mode did not work).

I found this handy list of boot options that allowed me to boot into safe mode and enable the Administrator account:
  • At the F10 prompt, I typed /SAFEBOOT:NETWORK to enable Safe Mode with Networking.
  • Once the machine booted, I used a cmd prompt as administrator, and typed:
  • net user administrator /active:yes

Other boot options that I noticed might be of interest:
  • /MININT - Starts the Windows Pre-installation Environment


I forgot my password, so here's a script...

gnome-keyring is what a lot of Linux / UNIX desktop environments rely upon in order to save passwords. Here's a script that dumps the contents of the gnome-keyring. It requires the gnomekeyring python module.

Special thanks to Michael Schurter for the majority of this code. I added two small improvements and deleted some dependencies.

#!/usr/bin/env python2

import getpass
import gnomekeyring

def dump_pass():
    gnomekeyring.unlock_sync(None, getpass.getpass());
    for keyring in gnomekeyring.list_keyring_names_sync():
        for id in gnomekeyring.list_item_ids_sync(keyring):
            item = gnomekeyring.item_get_info_sync(keyring, id)
            print '[%s] %s = %s' % (
                    keyring, item.get_display_name(), item.get_secret())
        else:
            if len(gnomekeyring.list_item_ids_sync(keyring)) == 0:
                print '[%s] --empty--' % keyring

if __name__ == '__main__':
    dump_pass()

Saturday, August 8, 2015

Forcing a Windows 10 upgrade in an iSCSI environment

If you boot Windows machines from iSCSI, you should know that the Windows 10 Upgrade doesn't load iSCSI drivers and dumps you into a recovery environment. However, there is a way to get around that.

Once the Windows 10 recovery environment (Windows PE) boots, use the Advanced Options to obtain a command line.

To start the iSCSI initiator:
wpeutil initializenetwork
net start msiscsi
iscsicli qaddtargetportal portal.ip.address.here
iscsicli listtargets
iscsicli qlogintarget iqn.xxx.xxx
iscsicli PersistentLoginTarget T * * * * * * * * * * * * * * * 0 (that was fifteen * characters)
iscsicli ListPersistentTargets
iscsicli ReportTargetMappings
Once this is ready, you can manually install Windows 10:
 c:\$Windows.~WS\Sources\Windows\sources\setuperror.exe
For reference, here are the hidden upgrade folders and their meanings:
"Windows.WS = Windows Server Folder"
"Windows.BT = Windows Backup Files"
"Windows.Old = Windows backup files"

Have fun!

Sunday, August 2, 2015

How to delete the recovery partition in Windows 8 / 10

I purchased a (cheap) Asus X205TA laptop that has an integrated "32 gigabyte" SSD. Unfortunately, this computer came with a 10 gigabyte recovery partition which lopped off one third of the computer's storage capacity. One third of the device's drive capacity for a function that I will probably use once during the laptop's life? There must be a solution.

UPDATE 8/3: Apparently in Windows 8/8.1, Acer and other vendors use a tool called "WIMBOOT" which uses files stored in the recovery partition to boot your main system (although, surely your machine doesn't require 10gb of data to start up every time). The solution? Upgrade to Windows 10, and "Back up system files" as described below.

DISCLAIMER: BACK UP YOUR DATA FIRST. The following directions may result in irreversible filesystem corruption, particularly if you make a typo. I recommend using any commercial or free drive imaging software prior to following these instructions, and offer no support or warranty if disaster strikes.

Microsoft's documentation seems to indicate that it is possible to delete the recovery partition from your PC as an option when creating a USB recovery disk. However, this did not work on my machine. Fortunately, there is a solution; however, it involves a few "advanced" steps that include using a command-line tool called diskpart:
  1. Obtain a > 4GB USB drive
    Windows needs just under 4 GB to create a recovery drive for your machine. Technically, this step isn't required if you allowed your PC to automatically upgrade from Windows 7 / 8 to Windows 10 as you can use Microsoft's Media Creation Tool.

  2. Create a recovery drive
    Use the media creation tool above, or (even easier) type "Create a recovery drive" in the Start menu's search box. Make sure you choose "Back up system files" in Windows 10 when creating your recovery media.
  3. Boot from the recovery drive:
    1. Type "Change advanced startup options" in the Start menu search area
    2. Under "Advanced startup," press the "Restart Now" button.
    3. Click the "Use a device" icon
    4. Select the option that corresponds to the removable device that you made into a recovery drive to boot
  4. Obtain a command line and launch diskpart:
    1. Choose your keyboard layout
    2. Under "Choose an option," select "Troubleshoot"
    3. Under "Troubleshoot," choose "Advanced options"
    4. Under "Advanced options," choose "Command Prompt." A big black box appears.
    5. Type diskpart and hit enter.

    NOTE: The following directions assume that the "Rescue" and "Primary" partitions are next to each other; i.e. Rescue is Partition 4, and Primary is Partition 3. If this is not the case, you should STOP HERE and use something like Partition Magic or gparted to do the job.
  5. Delete the recovery partition
    1. type: list disk
    2. The resulting list will probably have your system's disk as "Disk 0." Look at the Size column to figure out if it corresponds to the size of your internal drive.
    3.  type: select disk x, where X is the disk # of your system's internal drive.
    4. type: list partition. Look at the "Type" column for a "Rescue" partition (the size should be ~ 10 GB).
    5. type: select partition x, where X is the partition # of your system's rescue partition.
    6. WARNING: the following step will delete the partition. Double-check your work.
      type: delete partition
  6. Resize the "primary" (system) partition and filesystem
    1. Consult the results from list partition and type: select partition y, where Y is the partition # that corresponds to the "Primary" system partition (it should be the biggest partition, by size).
    2. type: extend
    3. type: extend filesystem
    4. type: exit and hit enter, then type exit again. The command prompt window should close.
    5. type: list partition. You should see that your "Primary" partition is now much bigger. Hooray!
  7. Reboot into Windows
    1. Under "Choose an Option," click Continue (boot into Windows)

Tuesday, June 16, 2015

Linux BTRFS newbie thoughts: rolling back changes made to the root subvolume

If you're more intelligent than me, when you started using btrfs, you probably planned out how you are going to manage your data within a hierarchy of snapshots before using a btrfs filesystem. Let me state this more plainly, using more directive language: when using btrfs, consider mounting the root subvolume (subvolid=0) somewhere (e.g.: /media/btrfs/volume_name), for the sole purpose of managing snapshots.

Unfortunately, I wasn't that smart. I mounted my "root" btrfs volume on /backup, and willy-nilly copied files into the top level of /backup. Only later did I realize I wanted to use snapshots to allow granularity in changes made to /backup, so I created a directory /backup/snapshots/before_modification with the commands:
% mkdir /backup/snapshots
% su
# btrfs subvolume snapshot /backup /backup/snapshots/before_modification
I then made changes to the files inside of /backup, and messed something up, and then wanted to revert my changes. Okay, no problem; the btrfs system admin guide says that all I need to do is to mount the snapshot in question:
# btrfs subvolume list /backup
ID 666 Gen 5902 top level 5 path snapshots/before_modification
# umount /backup
# mount -o defaults,subvol=666 /dev/sdc1 /backup
Well, great! That worked fine; my ugly changes don't appear in /backup any more. But my btrfs filesystem is undoubtedly saving changes that I don't want to keep; these changes are associated with the 'root' subvolume (id=0). Surely, I can now delete these changes, right? (For the truly curious, you can measure the space used by a snapshot after you enable quota management). Can't I simply:
# btrfs subvolume delete 0 /backup
NOPE! Documentation says that the only argument to 'delete' is a path, and doesn't involve a subvolume ID. Uhhh... now I'm stuck. At this point, my puny brain imploded, and I had to visit #btrfs on freenode, where user darkling helped me out.

The solution? To release all of the space held by the root subvolume:
# mount -o defaults,subvol=0 /dev/sdc1 /media/btrfs/backup
# cd /media/btrfs/backup
# rm -rf dir1 dir2 dir3 dir4 ...
From there, I selectively deleted all subdirectories inside of /media/btrfs/backup -- and carefully excluded the /media/btrfs/backup/snapshots directory.

Note: even if you think you're clever, and decide to change permissions of the /media/btrfs/backup/snapshots directory itself, remember that the rm -rf command will still delete any accessible subdirectories with write permissions. In other words, DO NOT rely on permissions of the snapshot directory alone to keep your files safe.

And now, I have a directory structure very similar to that described in the btrfs system administration guide...







Tuesday, June 2, 2015

Poor Windows 7+ application performance, especially Skype? Consider IE 10 WebCache settings

I don't even use Internet Explorer, so I was very surprised when my profile included hundreds of megabytes of junk inside of my %HOMEPATH%\AppData\Local\Microsoft\Windows\WebCache directory.

I attempted to modify permissions of this directory so that no data could be stored inside, and that caused terrible performance issues and errors reported in the Windows Application event log reported by taskhostex which was running the Wininet Cache task object.

The solution was to run inetcpl.cpl and click on the General tab -> Settings -> Caches and databases tab, and then uncheck "Allow website caches and databases."

After that, I launched Process Explorer and killed taskhostex, which had open file handles to several files within this directory. I then deleted those open files.

I then removed write permissions on the WebCache folder for my user, SYSTEM, and for Administrators.

This allowed Skype and other applications to run without writing data to WebCache.

Saturday, May 30, 2015

Tips / simple tool to help me play DVDs on my Fire TV stick

Now that we have an Amazon Fire TV stick, I wanted to play DVDs without having to have more equipment plugged into my TV. I created the following tool that helps back up my DVDs to a folder on my server. Check out the project here:

https://github.com/baitisj/dvd_auto_mirror

The script simply opens the DVD drive tray, and waits for you to shovel DVD after DVD into the drive.

I also wanted an easy way of sharing this directory with my Fire TV stick. I sideloaded Kodi, and discovered that HTTP shares don't work, but WebDAV does. My lighttpd.conf file:

server.port        = 80
server.username        = "http"
server.groupname    = "http"
server.document-root    = "/var/www/"
server.errorlog        = "/var/log/lighttpd/error.log"
server.modules += ( "mod_webdav" )
dir-listing.activate    = "enable"
index-file.names    = ( "index.html" )
mimetype.assign        = (
                ".html" => "text/html",
                ".txt" => "text/plain",
                ".css" => "text/css",
                ".js" => "application/x-javascript",
                ".jpg" => "image/jpeg",
                ".jpeg" => "image/jpeg",
                ".gif" => "image/gif",
                ".png" => "image/png",
                "" => "application/octet-stream"
            )

$HTTP["url"] =~ "^/dvds($|/)" {
    webdav.activate    = "enable"
    webdav.is_readonly = "enable"
}
 I set up a symlink from /var/www/dvds to my archive directory. Done!

Friday, May 22, 2015

Fix Google Search (Missing terms?) HOWTO

A little while back, Google Search realized that most humans are stupid, and it started to think that it was pretty clever by including search results that didn't include all of the supplied search terms. Finally, they have added the "verbatim" option:

http://i.imgur.com/HSpKQDgl.png

It turns out that you can change your default search engine to "Google Verbatim" to restore the "old" behavior of Google:

http://mycroftproject.com/search-engines.html?name=google+verbatim

Thanks so much for doing that in the first place, Google.
Missing: You're welcome.

Tuesday, May 12, 2015

pyWebDav <-> Windows 8.1: one-line file sharing, after a couple of registry tweaks

How often do you want to share a directory of files on one (*nix) system, and mount it as a network drive on a Windows host, with minimum hassle? For me, this is a common occurrence. After a couple of adjustments to my Windows 8 machine, I can do this after installing the pywebdav module, making for a one-liner that creates a temporary high-throughput network share that can be mounted as a drive in Windows.

The annoying part of this trick is that it takes a couple of registry tweaks, and this may affect other resources shared through WebClient, notably, SharePoint.

The easy part of this is installing the pyWebDav Python package, and running a one-line command to start a WebDAV server. Install this (in BSD we type pkg install py27-PyWebDAV). The -D option specifies the directory to share, and I like to know what the program is thinking (verbose with -v option):
davserver -D directory_to_share -n -H server_ip_address_here -v
davserver needs the IP address of the network resource that it will use to listen for connection requests. You can test the server using the loopback address and a webdav client installed on the same machine.

Now for the more annoying part: registry changes, with some commentary. On a Windows machine, navigate to the following subkey in regedit:

HKLM/System/CurrentControlSet/Services/WebClient/Parameters

Modify these values:

SupportLocking=0x0 (default=0x1) 
REQUIRED for interoperability with pyWebDav. 
  • davserver's -J parameter appears buggy / nonfunctional with Windows 8, so we have to modify the registry.

FileSizeLimitInBytes=0xffffffff (or whatever, default is 50 000 000d)
Optional, but you should change this.
  • The default value of this is only 50 megabytes, which is really useless for ISOs or whatnot. I recommend 4.2 gigabytes, which is what this value represents. A bigger value would be preferable. Fix this, Microsoft! We've moved on to use terabyte devices.

BasicAuthLevel=0x2

Optional change, useful if you add basic authentication (which pyWebDav supports)
  • 0 - Basic authentication disabled
  • 1 - Basic authentication enabled for Secure Sockets Layer (SSL) shares only (default)
  • 2 or greater - Basic authentication enabled for SSL shares and for non-SSL shares
Now that you've made these changes, you can restart the WebClient service. Launch cmd with administrator privileges and type:
net stop  WebClient
net start WebClient
 Mount the share (note that you'll need to do this in a cmd shell WITHOUT administrator privileges for your user to see the drive):
net use * http://server_ip_address:8008/
You can also map network drives using a more GUI method.

Sweet!

USB audio disconnect woes in FreeBSD with fstat: tales of a voltage-sensitive Topping TP30

I have a very nice "class T" Tripath audio amplifier: the Topping TP30 (MK I -- hopefully they fixed the MKII) with a very convenient integrated USB audio card. However, this device is VERY sensitive to line voltage. For example, when plugged into the same circuit as, say, a refrigerator, the motor spark is enough to pop the speakers and cause the device to perform a USB reset.

In the case of FreeBSD, this causes everything to come to a miserable, grinding halt. dmesg gives me repeated warnings of failure:

pcm2: unregister: mixer busy
pcm2: Waiting for sound application to exit!
pcm2: unregister: mixer busy
pcm2: Waiting for sound application to exit!
pcm2: unregister: mixer busy
pcm2: Waiting for sound application to exit!
pcm2: unregister: mixer busy
pcm2: Waiting for sound application to exit!
pcm2: unregister: channel pcm2:virtual:dsp2.vp0 busy (pid 11413)
pcm2: Waiting for sound application to exit!
pcm2: unregister: channel pcm2:virtual:dsp2.vp0 busy (pid 11413)
pcm2: Waiting for sound application to exit!
pcm2: chn_write(): pcm2:virtual:dsp2.vp0: play interrupt timeout, channel dead

Before I realized that fstat worked on character devices, I had to manually fish around for processes to kill, in order to give the USB audio subsystem the breathing room required for it to reset the audio device. Interestingly enough: if a process that was holding onto the mixer releases the corresponding /dev/mixer device, but then restarts quickly enough and grabs control of the (now defunct) mixer, the sound card still won't reset.

Below, find the one-liner script that goes out and kills every process that holds the audio devices open. I have to execute this twice in a row to kill off XFCE's mixer process until the USB audio card has time to fully reset:

fstat /dev/dsp* /dev/mixer* /dev/audio* | tail -n +2 | awk '{print $3}' | xargs sh -c 'sudo kill -9 $0 $*'

Tuesday, March 17, 2015

Rid thyself of that pesky PulseAudio daemon in Arch / Manjaro Linux

I have a very lightweight system -- a laptop from 2008 -- that usually works quite well until PulseAudio gets in the way of things.

Here are some tricks to disable PulseAudio:

  1. Edit /etc/pulse/client.conf and add the line:
    autospawn = no
  2. If you want to go "the whole nine yards" and uninstall PulseAudio entirely, you can build a Gnome package that does not depend on PulseAudio:
    yaourt -S gnome-settings-daemon-nopulse
  3. Next, you can entirely remove packages that depend on PulseAudio:
    pacman -R pulseaudio pulseaudio-alsa lib32-libcanberra-pulse lib32-libpulse manjaro-pulse xfce4-volumed-pulse skype
    Note that Skype 4.3 requires PulseAudio. Therefore, you have little choice but to remove it.

Sunday, February 8, 2015

PC-BSD: Running QT applications from within a Warden "Ports" jail (for X11 applications)

I've been running PC-BSD 10.1.1 with great success. However, there are a few "bleeding edge" applications that I want to compile and run; most notably, Anki, my favorite flash-carding application / learning aid.

I started out by creating a "Ports Jail (insecure, allows running X applications)," which automatically downloads and installs the FreeBSD ports tree (a large library of applications that can be compiled to run on a FreeBSD system). I installed a few packages (binary, precompiled programs), including the ssh daemon and sudo.

From there, I descended into the ports tree in /usr/ports/games/anki, and then I executed a recursive make command, to automatically compile everything
/usr/bin/nice -n 19 make DISABLE_VULNERABILITIES=yes -DBATCH
This worked swimmingly, and a subsequent make install successfully integrated Anki into my jailed environment. However, a few issues remain:

  1. X11 over SSH:  ssh -Y username@ports_jail does NOT forward X11 correctly. Launching any X11 program results in the following message:
    X11 connection rejected because of wrong authentication.
    Error: Can't open display: localhost:11.0
    If you're SSHing in via a shell within your computer's X11 environment, this can be rectified with:
    export DISPLAY=":0.0"
    However, this doesn't really fix the ssh X11 forwarding problem, and I'm still at an impasse.
  2. QT doesn't render correctly. When launched without environmental variables, I get the following results:
    • A number of X11 errors when launching from the command line:

      QNativeImage: Unable to attach to shared memory segment.
      X Error: BadDrawable (invalid Pixmap or Window parameter) 9
      Major opcode: 62 (X_CopyArea)
        Resource id: 0x0
      X Error: BadDrawable (invalid Pixmap or Window parameter) 9
      Major opcode: 62 (X_CopyArea)
      Resource id: 0x0
    • The initial screen is rendered improperly: gray windows, weird artifacts. 
    To solve, modify the QT_GRAPHICSSYSTEM environment variable:
    QT_GRAPHICSSYSTEM=native anki
I'm going to post my experiences to the PC-BSD forums, and see if some folks have some ideas about how to fix the issue.

Saturday, January 17, 2015

PC-BSD: FreeBSD 10.0 -> 10.1 upgrade using freebsd-update

I was having problems getting pc-sysupdate to work properly. For some reason, purging the pkg cache and forcing updates through pkg update -f resulted in package size mismatches, so I decided to try to upgrade my PC-BSD system manually. It took a while, but I think I figured out how to get it going.

First, I created a boot environment for my new upgrade:

beadm create fbsd_10.1_update
beadm mount fbsd_10.1_update /.updateStage


Next, I performed the freebsd-update on this new boot environment:

freebsd-update -b /.updateStage fetch
freebsd-update -b /.updateStage install


Eventually, the script finished its business. I then switched to the new environment:

beadm umount fbsd_10.1_update
beadm activate fbsd_10.1_update


All done, right? Here's what happened next. I rebooted the machine, and was dropped to a grub> prompt. What? beadm didn't handle grub gracefully?

Ugh. After digging, I figured out how to manhandle grub to boot my ZFS partition. The first trick, to save time, is to set an environment variable to the root filesystem. This saves a LOT of typing. Use "tab" completion to figure out what filesystems are available to you:

set P=(hd0,msdos2)/ROOT/fbsd_10.1_update/@/


Now, we can tell grub to load all the things (tab completion works with the $P expansion! brilliant!):

kfreebsd $P/boot/kernel/kernel
kfreebsd_loadenv $P/boot/device.hints
kfreebsd_module_elf $P/boot/kernel/opensolaris.ko
kfreebsd_module_elf $P/boot/kernel/zfs.ko
kfreebsd_module $P/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache
set kFreeBSD.vfs.root.mountfrom=zfs:tank/ROOT/fbsd_10.1_update


boot


Voila! System working again. But I still need to repair GRUB... after the system boots into the correct boot environment:

grub-mkconfig -o /boot/grub/grub.cfg

Unfortunately, that last step didn't actually fix my grub installation. I'm puzzled as to how to fix it. But for now, at least I can get the machine running.

Monday, January 5, 2015

Reward dollars: Macy's, Nike, Brooks Brother's, and Barnes & Noble gift cards 10% off on American Express rewards portal until 1/15

Get the most out of your Blue Cash cards. American Express has select gift cards on sale now:
Barnes & Noble Gift Card
Barnes & Noble Gift Card
Sale: 45.00 Reward Dollars
Sale: 45.00 Reward Dollars

Sale: 45.00 Reward Dollars
Sale: 45.00 Reward Dollars
Offer ends 01/15/2015 or while supplies last