Building immersive sound walks in 2019

Me again.

What happened to the Internet? Why is everything fun and creative that I remembered and was sick of around 10 years ago all of a sudden gone? Flash has been my nemesis for years, and I jumped on every opportunity to crack a moderately funny Flash joke back in my creative coding/interaction design days, but its discontinuation has also left a gaping hole of sadness in the development of quirky, open, fun multimedia experience websites.

All I wanted to quickly build was a web app of some sorts that plays media based on your location. Let’s dumb it down, and make it sound only, and focus on building a sound map. This doesn’t even need to be location-based. Just a site with a 2D representation of territory, and click or tap on a particular feature, play an audio track.

So what’s out there?

I just discovered Echo.XYZ and was hopeful. Their website sounds like a bunch of fun, creative, energetic people took the initiative and created a solution for their particular problem and decided to share it with the world.

We came together looking for a way to create a GPS-triggered audio tour in Hanoi in 2013, and ended up deciding to create our own platform.

I immediately feel invited when they tell me to create breath-taking GPS-triggered tours wherever I am. Sounds like me. Tops.Screen Shot 2019-01-10 at 6.35.14 PM.png

I have all I need to test this out. An image of my sound recording and a sound (converted to m4a). I find the location on a map and put everything nicely together, but I am confused how to preview my work. After all, I lose all the work I have done. I give it another go and then realise I probably need to download the app. An iOS device at hand I download the app and fail to login with my credentials the first time. Weirdly, the second times it works. The bugginess continues when I select my walk. It is great that I can download all data while on Wi-Fi to avoid data charges during my walk, but when I want to start, the app crashed. Also the second, third, and fourth time.

img_3771

I give up. This could have been too good as a prototyping solution. Unfortunately too buggy.

Extracting Closed Caption from Youtube Video

“Mom, I can’t, I need to transcribe a Youtube video” –
“No, you don’t. And now come down and check out the zine I made out of your Snapchat comments with the Python NLTK!”

What does mom mean?
Of course, no one transcribes a Youtube video nowadays when the machines already have done so. Here are the easy steps to download the (automatic) Closed Captions on anyone’s video to a text file.

You need:
A browser with Developer Tools such as Firefox or Chrome.
A word processor or editor like Notepad, TextEdit or Word.

You will:
Use the Network Monitor of Firefox Developer Tools to find and download the Closed Caption File generated by Youtube.

  1. Open a Youtube video of your choice, play and pause.
  2. Open the Tools… Web Developer Console… Network.Screen Shot 2018-12-04 at 12.42.54 PM.png
  3. Click on the CC-Closed Caption button of the Youtube Video.Screen Shot 2018-12-04 at 12.41.46 PM 1.png
  4. A file starting with the name timedtext should appear in the Network console window.
    Screen Shot 2018-12-04 at 12.43.09 PM.png
  5. Right-click on the file and select Open in New Tab…
    Screen Shot 2018-12-04 at 12.43.34 PM.png
  6. From here go to File…Save Page As…
    Screen Shot 2018-12-04 at 12.43.56 PM.png
  7. Rename the .xml file to .html and open it in your favourite text editor.
  8. To remove the line breaks from the file, you can use for example the Find and Replace function in Word and replace the Paragraph Mark (^p) with a Space Character. Screen Shot 2018-12-04 at 12.46.23 PM.png

That’s it.
Add some punctuation to your own liking.
The raw final Rosa Park’s Speech from 1998 would look like this in the end, would need some editing but is a good basis for a transcript.

thank you so very much and they’re happy to be here this is a wonderful evening and we have so many beautiful people here and both youth as and adults and I’m very glad that we came to this program and I’m hoping that we will continue to do all that we can to help our youth because they are our future and we have a wonderful leadership I have a sure dr. cater would be happy to see this fruit if he was only here because he believes in freedom and and love for all human and of course all of you here are just wonderful with little ones as well as our own and I’m always grateful to dr. Takahashi to talk of okay so talk a hats a fine if I’m not just right but anyway she has been so a wonderful worker because most of my books I think all of my boots on do to her now translated to japanese see another friend here from I’ve been knowing him all of his life that she’s a little boy Victor too so all I can say is I feel over and over joyed with the wonderful bowing of love that I haven’t received here today this evening and I’m hoping that we can continue with our programs of the rules and Raymond Fox Institute for self development and the pathways to freedom will continue to work with young people and continue to show them the way to be good citizens and have some spiritual awareness in their lives and keep themselves really on the way to freedom and equality for all of us thank you and God bless let’s takes once again let’s take miss mrs. Rosa Parks we’re telling tonight’s video you

Python3, Django

I am running macOS High Sierra 10.13.6 and attempting to install the latest Django 2.1.2. With 2.7 being the standard Python on my machine and a vague memory of basically bricking my Mac by changing anything around Python versions back in the days I am resorting to virtualenv to run python3 and install the latest version of Django without risking a mental breakdown.

When I follow any instructions that tell me to

pip install Django==2.1.2

I get the following error:

Could not find a version that satisfies the requirement Django==2.1.2 (from versions: 1.1.3, 1.1.4, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.6, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.6.8, 1.6.9, 1.6.10, 1.6.11, 1.7, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6, 1.7.7, 1.7.8, 1.7.9, 1.7.10, 1.7.11, 1.8a1, 1.8b1, 1.8b2, 1.8rc1, 1.8, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.5, 1.8.6, 1.8.7, 1.8.8, 1.8.9, 1.8.10, 1.8.11, 1.8.12, 1.8.13, 1.8.14, 1.8.15, 1.8.16, 1.8.17, 1.8.18, 1.8.19, 1.9a1, 1.9b1, 1.9rc1, 1.9rc2, 1.9, 1.9.1, 1.9.2, 1.9.3, 1.9.4, 1.9.5, 1.9.6, 1.9.7, 1.9.8, 1.9.9, 1.9.10, 1.9.11, 1.9.12, 1.9.13, 1.10a1, 1.10b1, 1.10rc1, 1.10, 1.10.1, 1.10.2, 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7, 1.10.8, 1.11a1, 1.11b1, 1.11rc1, 1.11, 1.11.1, 1.11.2, 1.11.3, 1.11.4, 1.11.5, 1.11.6, 1.11.7, 1.11.8, 1.11.9, 1.11.10, 1.11.11, 1.11.12, 1.11.13, 1.11.14, 1.11.15, 1.11.16)
No matching distribution found for Django==2.1.2

So here we go. I got virtualenv (pip install virtualenv) installed and got python via homebrew that I can run with python3 command. So I make a directory like /Users/merglind/Project and get going.

virtualenv myamazingrainbowenv -p python3

So far so good.

source bin/activate

Now the little terminal thingie should feature something starting

(myamazingrainbowenv) Users/Merglind:Django

So now that I am in my jolly virtualenv with Python3 and all, I run

pip install Django==2.1.2

I get a terminal screen happily filling up with chars of all sorts until it gleefully prints the magic words:

Successfully installed Django-2.1.2 pytz-2018.5

Geotagging WordPress Posts

I believed what I was after was reasonably simple and straightforward. I want to associate a range of content (text, image, audiovisual media…) to geolocations and create a website that allows geodata-based storytelling. Basically, I want a map that displays a range of geotagged data ranging including simple descriptive markers, blog posts including audiovisual data and sensor data, eventually, include live IoT data.

One inspiration, even if a little clunky to use and technically outdated because it is running Flash is Radiant Copenhagen. It appears to be combining PM Wiki and Google Maps.

Screen Shot 2018-10-03 at 8.27.21 PM.png

Figure 1: Radiant Copenhagen (Note the watermark on the map possibly caused by an expired Google Maps API key)

Locus Cast is a more recent project and features a sound map containing links to geo-tagged audio live streams. Again, Google Maps is used.

Screen Shot 2018-10-03 at 8.33.35 PM.png

Figure 2: Locuscast. Screenshot of the Sound Map showing icons of open microphones.

Screen Shot 2018-10-03 at 8.37.10 PM.png

Figure 3: Locuscast. Screenshot with content displayed on top of map when an open microphone is selected. This view includes live streamed audio.

Some of my content is already hosted on WordPress. Since WordPress is a popular, open and well-maintained Content Management System I started to look into WordPress as a means to organising my content. Because post types can be anything from a simple URL, image, longer blog post containing media etc, categorized and tagged I thought I could associate every post with a geotag and group/display them on a map. The second idea was to create categories for my posts (e.g. associated with certain areas of my fieldwork) and link these to a certain location, so that content about a specific location can be easily filtered and displayed together.

I am, for example, using a WordPress blog to collect fieldnotes and I wanted to attach a location to that post, then create a page where posts can be viewed/sorted by geolocation on a map interface.

And so the journey begins.

Many geotagging plugins, especially lightweight solutions, appear abandoned/not supported anymore. I am tending more to use FOSS software using something like Open Street Maps, avoiding Google Maps, using API keys as much as possible.

A couple of Plugins I have installed so far, with no success yet.

WP-Geometa was the most promising, as it is open source and claims to “add real spatial support to WordPress. All I read about it sounds great and also this talk suggests that this plugin is just the beginning of the opportunities for marrying WordPress and GIS. Unfortunately, after install, I have no idea what I am looking at. I haven’t worked with wp meta before and some of the GIS lingo is beyond my head, even though I have worked with basic spatial data for a while now. There is no FAQ because no one has asked any questions yet, but honestly, don’t know where to start…

Geo My WordPress offers advanced geolocation, mapping, and proximity search plugin. Geotag post types and BuddyPress members, and create advanced proximity search forms. So far so good, but the plugin appears to brute-force any location I select to an address. Since my fieldwork is located in the bush I need exact Lat/Lon locations and not address proximations.

Leaflet Map was a hot candidate because I have some experience with Leaflet as a highly versatile and mobile-friendly map solution. The plugin allows associating a post with a variety of maps via shortcodes and the Shortcode Helper is very useful. But this seems more directed at posts that are looking for beautiful, versatile maps within a post, rather than associating a post with a location.

The OSM plugin looked great at first but didn’t really work so far: “Customize maps in your post, pages and widgets. GPX, KML and more. The easy way to map!” says the blurb, but usability is clunky. I can add markers to a map in a post and apparently I can show maps with those markers and filter them, but this did not end up working. This tutorial is titled “how to geotag posts and display map with geotags” but the instructions are a bit vague and unclear.

Next up: Leaflet Map Markers. Calling itself the “The most comprehensive & user-friendly mapping solution for WordPress” I can’t wait to see some results. Straight upon install, the plugin offers me a 30-day free trial version of the Pro version. Meh. Immediate downer. I am not sure which features are part of the freemium and at what point I will be tempted to spend money on something that I falsely believed to be a reasonably simple integration. Scrolling through all the functionality I feel a bit like crying. I can add maps with layers and markers but I am not sure if these can be associated with posts. Also, a very orange rectangle keeps reminding me to upgrade to the Pro version. I am staying a Noob for now.

OSM categories looked promising but hasn’t been updated for 6 years. I install it anyways, just to find out I have no idea how/if it would work as the documentation is very poor. Abort, abort. MapPress is, similarly to Leaflet Map just a good tool for embedding maps into posts, ideally for businesses. Geolocation works with Google API.

After all this, I am tempted to just set up my own MySQL database with some basic content and data and manually display the locations on a map. Or use something like Django. Seems like more work at first but all those options so far appear way too much of an overkill for what I need.

And then there are those promising Google Maps solutions like WordPress Google Maps the free version and the Pro version that might or might not technically do what I want but is not open source and uses Google Maps which I intended to avoid for this work.

Getting started with Scrapy on Mac OS X High Sierra

This post documents installing Scrapy on a MacBook Pro (13-inch, 2017, Four Thunderbolt 3 Ports) running macOS High Sierra. The TL;DR setup steps would be:

  1. sudo easy_install pip
  2. sudo pip install virtualenv
  3. Mkdir a directory to be used with scrapy and activate virtualenvsource ENV/bin/activate
  4. pip install scrapy into this virtualenv
  5. Get started with scrapy scrapy startproject craigslist by following a tutorial e.g. scraping Craigslist

The long, real-time documented version including mildly funny commentary can be found below.

I am on High Sierra, 10.13.5, nowadays and my laptop is so futuristic it has a touchy touch bar that no one has found a really good use for yet, and the buses are so fresh that no contemporary device can physically connect to it.
However, today I am trying to scrape some web with this amazing shiny piece of aluminium.
After having worked with lots of manual scraping techniques from the command line, like urrlib2 and Beautiful soup and whatnot, today I will give Scrapy a go.
Logo of Scrapy showing a scrapy spatula
The cute scrapy spatula icon gives me hope, that this “open source and collaborative framework for extracting the data you need from websites” really works “In a fast, simple, yet extensible way.” So here we go.

  1. This computer is pretty much naked when it comes to any useful command line tools so to install Scrapy I need pip first.
    sudo easy_install pip
    Bummer the fingerprint reader from the future doesn’t work on the command line and I actually need to type out my very safe and strong password. A couple of lines printed in my window. So far so good.
  2. pip install scrapy
    A couple of lines dumped in my terminal suggest this is working out until I get some red warnings around pillow, nose and tornado. Dependencies, here we come…
  3. sudo pip install pillow
    sudo pip install nose
    sudo pip install tornado
    Unclear if that just worked? – No, not really.
  4. Cannot uninstall 'six'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

OK, fail, back to start. I take the time and actually read a bit more. It is recommended to install scrapy in a virtual environment. Well here we go.

  1. sudo pip install virtualenv
    So far so good, let’s just ignore the warnings for now.
  2. Looks good. I got my project folder /Users/.../scrapy so this is where I create my virtual environment ENV by virtualenv ENV
  3. I activaye this environment with source ENV/bin/activate
  4. Here we go. I am in my virtual envirnoment. Hurrah. So what happens when I install Scrapy in here? pip install scrapy
    My terminal joyously floods with words and vintagely appealing ASCII progress bars until we gleefully end up having installed all kinds of dubious packages by the names of Automat-0.7.0 PyDispatcher-2.0.5 PyHamcrest-1.9.0 Twisted-18.7.0 asn1crypto-0.24.0 attrs-18.1.0 cffi-1.11.5 constantly-15.1.0 cryptography-2.3 cssselect-1.0.3 enum34-1.1.6 functools32-3.2.3.post2 hyperlink-18.0.0 idna-2.7 incremental-17.5.0 ipaddress-1.0.22 lxml-4.2.3 parsel-1.5.0 pyOpenSSL-18.0.0 pyasn1-0.4.3 pyasn1-modules-0.2.2 pycparser-2.18 queuelib-1.5.0 scrapy-1.5.1 service-identity-17.0.0 six-1.11.0 w3lib-1.19.0 zope.interface-4.5.0
  5. I follow the Getting Started tutorial scrapy startproject craigslist and end up getting a message that confirms the successful creation of the Craigslist Scrapy project.

Stream addresses QGIS

First steps towards some data wrangling in QGIS. Or any other software will do.

I have a vector layer (.gdb) of historical stream data and I want to know the current addresses where the stream layer crosses a property.

I have downloaded both open property and address datasets (there’s probably a combination of both which would make this much easier) and added them on top of an OSM layer, using QGIS 16.2.3 Mac OSX. So far so good:

Here is a Screenshot of how the data looks at the moment:

Streams 1936 .gdb  – blue [thanks WCC]
NZ property titles .shp – teal [thanks LINZ]
NZ street address (2) .shp – green  [thanks again, LINZ]Screen Shot 2017-07-13 at 4.53.11 pm

At this stage this will be a two-step process.
First, identify all properties that overlap with a Stream feature.
Then, identify the address(es) of this properties and export them in a somewhat readable form (.csv). So the feature attribute of nz-street-address-2 that I want for now would be full_add_2. (e.g. 2/11 Arlington Street, Mount Cook, Wellington).

The stream data doesn’t include stream names so I might need to add this as a field to the dataset that in the end I would have something like:

Papawai Stream 20 Street Street, Suburb, Wellington
Papawai Stream 21 Street Street, Suburb, Wellington
Papawai Stream 21b Street Street, Suburb, Wellington
Papawai Stream 21 This Avenue, Suburb, Wellington
Some Other Stream 21 That Avenue, Suburb, Wellington

 

 

Copy all videos please

Got some video files in a bunch of folders and want to collect them in a new one?
Here’s a bash script that moves files ending with .mov of subdirectories into a folder called success.

find ./ -name '*.mov' -exec cp -prv '{}' 'success' ';'

Working on a forked repo

I found this wonderful repository for adding Photos to a leaflet layer.

The original project by Github user turban appears abandoned but user HeikkiVesanto has forked the repo and added neat functionality, the use of a local directory for photos which allows a complete self-hosted solution (the original project used Instagram and the [also abandoned] Picasa).

To ensure longevity of my project I need a self-hostable solution, however, when I fork the repo (even when on the desired forked branch) I only get the original author’s work and not the more recent project. The network diagram shows that the version I forked is older and misses some work from other contributors.

Screen Shot 2017-04-26 at 4.12.42 PM

So I forked the repo and then cloned it to my local machine.

After some trial and error it was a quick fix to actually switch my local repo to the other author’s more recent fork:
Making sure I’m on the correct branch (in this case gh-pages)

git checkout gh-pages

Then pull the repo of the author:

git pull https://github.com/HeikkiVesanto/Leaflet.Photo gh-pages

and then push those back to my Github with

git push origin gh-pages

And

git status

happily returns

On branch gh-pages
Your branch is up-to-date with 'origin/gh-pages'.
nothing to commit, working tree clean

And the resulting graph shows that my project is now ahead of the original and in tune with the most recent changes:

Screen Shot 2017-04-26 at 4.15.10 PM

Headless raspberry pi MQTT broker

Three simple steps that took me weeks to figure out.
I want to run my raspberry pi AP headless, feeding off a 10400 mAh battery out in the wild while juggling the odd MQTT jabbering from my little Wemos D1s busy with making sense of their environment.
Because I’m hanging the Pi up high in a tree it might be nice to have ssh access to figure out what’s going on over the air.
So how did I finally come to a solution?

1. I enabled ssh in raspi-config and set a new password (but honestly only, after the CLI forced me to…. security in the bush… meh… “No, that one is too short. No, that one is too simple” CLI kept saying after I finally found something complex enough for it to accept and simple enough to remember for at least as long that I can find a pen and note it down.

2. While in the raspi-config I found the option to wait for network until boot – ha! That’s all. Raspi by default wants to boot as quickly as possible. “Slow down, and wait for my network” I said.

3. Start the broker I added the simple, slick line

mosquitto -d

to

etc/rc.local

before the exit 0 line so it runs at boot.
Woo! That was easy.
Now: lunch.