Zum Hauptinhalt springen
Chris’ wirre Gedankenwelt

Moving around darktable managed photos

Under some circumstances, it's necessary to move photos from one file system structure to another. In my case, I moved my old photos from my notebook hdd to a 2TB external device. With this, I also want to start with a clean database for my new photos.

In my example, I moved the photos from /home/chris/Photosto /media/EXTERNAL/Photos2011/Photos. Have a look at my last blog post . Within the Photosdirectory I have several sub folders, each representing one film roll. So, /home/chrisis the base directory of my Photo directory.

Assuming, you have moved your photos as well as the database and config files to /media/EXTERNAL/Photos2011, you now have to adjust the paths to your film rolls in library.db.

Lets have a look into library.db: (make sure, darktable is currently not running)

$ cd /media/EXTERNAL/Photos2011/.config/darktable
$ sqlite3 library.db
sqlite> .schema film_rolls
CREATE TABLE film_rolls (id integer primary key, datetime_accessed char(20), folder varchar(1024));
sqlite> select * from film_rolls;
1|2011:09:08 11:16:16|/home/chris/Photos/20110904_XXX
2|2011:09:08 11:17:46|/home/chris/Photos/20110908_YYY
[...]

As you can see, the table film_rollscontains an id, the last access time and the folder - which is the base directory of this film roll. If you want to, you also could have a look into the table images. Most important in it is the foreign key film_id, which refers to the film roll of the specific photo.

However, you just have to adjust the paths of the film rolls and leave the literal photos untouched.

The change in the database should result in something like:

$ sqlite3 library.db
sqlite> select * from film_rolls;
1|2011:09:08 11:16:16|/media/EXTERNAL/Photos2011/Photos/20110904_XXX
2|2011:09:08 11:17:46|/media/EXTERNAL/Photos2011/Photos/20110908_YYY

I just wrote a few lines of python to do this move_darktable_photos.py . This small script runs on python >= 2.6, build with sqlite3 support.

Just call it:

$ ./move_darktable_photos.py \
--library /media/EXTERNAL/Photos2011/.config/darktable/library.db \
-o /home/chris \
-n /media/EXTERNAL/Photos2011
Autor
Chris Glaubitz
Configuring, coding, debugging computers for a living. Riding bikes for fun.

Starting darktable isolated

Sometimes, e.g. if you moved your data to an external device which is not permanently available, or even if you just want to use separated databases for managing your photos, it is useful to to start darktable isolated. It is be possible to launch darktable with the --libraryparameter, to force darktable to use a specific database file:

$ darktable --library /PATH/TO/library.db

However, with this approach your "default" config files and thumbnail cache is edited as well. For me, that's annoying.

If you start darktable in an isolated home directory, each database comes with its own config files and image cache. Just copy .config/darktableand .cache/darktableto your external device (the library file is located in .config/darktable/library.db):

$ cp -rp ~/.config/darktable ~/.cache/darktable /media/EXTERNAL/Photos2011/

Take care, to update the path to your photos in library.db(film_rolls is the relevant table). I will write a few sentences about moving darktables photos in another post .

However, /media/EXTERNAL/Photos2011is the isolated home directory and you have kind of this structure:

.cache/
└── darktable
    ├── mipmaps
    └── mipmaps.fallback
.config
└── darktable
    ├── darktablerc
    ├── keyboardrc
    ├── keyboardrc_default
    └── library.db
Photos
└── 201112
    └── IMG0001.CR3

Remember, the paths to your images has to be updated!

Now you have to set the HOME environment variable to your new directory and start darktable:

$ HOME=/media/EXTERNAL/Photos2011 darktable
Autor
Chris Glaubitz
Configuring, coding, debugging computers for a living. Riding bikes for fun.

Neuinstallation Webserver (redpill)

Im laufe der nächsten Wochen 19.12.-01.01.2012 installieren wir unseren Server neu. Unsere Services werden in der Zeit also auf jeden Fall mal weg sein.

Aber als Weihnachtsgeschenk gibt es dafür ein schönes neues Arch :)

Autor
Chris Glaubitz
Configuring, coding, debugging computers for a living. Riding bikes for fun.

Scribus Vorlage Wandkalender 2012

Weihnachten steht vor der Türe und ich plane einen eigenen Wandkalender mit selbst geschossenen Fotos zu verschenken. Wie schon letztes Jahr lasse ich den Kalender bei Pixelspeed im A3-Querformat drucken.

Dafür habe ich ein Scribus Dokument erstellt, in dem nur noch die Bilder eingefügt und das daraus erstellte PDF hochgeladen werden. Wichtig ist, die Übergabestandards einzuhalten. Innerhalb des Scribus Dokuments gibt es eine Masterpage, in der mein Name noch ausgetauscht werden muss. Die eingefügten Bilder sollten in 300dpi vorliegen, werden ansonsten automatisch hochskaliert. Bei der Generierung des PDFs sollte darauf geachtet werden, die verwendeten Fonts zu integrieren und die PDF Version 1.3 zu erzeugen.

Achtung: Ich habe diese Vorlage noch nicht drucken lassen. Daher weiß ich noch nicht, ob die Fonts richtig rauskommen. Laut des Übergabestandards sollte ein Standard- und am besten keine freien Fonts verwendet werden. Daher habe ich mich für Lucida Grande entschieden, den ich direkt aus dem AUR installiert habe.

Über Feedback freue ich mich natürlich :)

Autor
Chris Glaubitz
Configuring, coding, debugging computers for a living. Riding bikes for fun.

Python and Flattr API v2

In Fiddled around with python-oauth2 and flattr I described how to access the Flattr API. Since this post, the Flattr guys relased a new API which is currently still beta but already useful working. However, at least after a fix of the access token process ;-)

So. What exactly changed? Almost everything, starting with a completely new and different authentication and authorization process.

I think, with v2 it's easier to develop apps. The API seems to be more straight forward. The process to get an access token is much simpler. I also like that JSON is used by default. Also the documentation is better than before, and very easy to understand and follow. What I really like, I don't need python-oauth2 as dependency any more. Everything is working with e.g. httplib2 .

First we have to send the User to Flattr, for authorizing our app. Therefor we have to include our client_id, which is the KEYof our app on https://flattr.com/apps :

https://flattr.com/oauth/authorize?scope=thing&response_type=code& client_id=KEY&redirect_uri=http://localhost

After the authentication, we will be redirected to http://localhost/?code=NEW_CODE.Your app should be registered now at https://flattr.com/settings/applications .

The next step is already to get the access token, using HTTP Basic Auth:

>>> http = httplib2.Http(disable_ssl_certificate_validation=True)
>>> http.add_credentials(key, secret)
>>> code=NEW_CODE
>>> params = {'code': code,
... 'grant_type': 'authorization_code',
... 'redirect_uri': 'http://localhost/'}
>>> ret = http.request('https://flattr.com/oauth/token', 'POST', j.dumps(params), headers={'Content-Type': 'application/json'})
>>> ret
({'status': '200',
  'content-length': '73',
  'strict-transport-security': 'max-age=500',
  'set-cookie': 'PHPSESSID=3hceaj0umb533cipg462briq64; path=/; domain=.flattr.com; HttpOnly',
  'expires': 'Thu, 19 Nov 1981 08:52:00 GMT',
  'vary': 'Accept-Encoding',
  'server': 'lighttpd',
  'connection': 'close',
  '-content-encoding': 'gzip',
  'pragma': 'no-cache',
  'cache-control': 'no-store',
  'date': 'Sun, 20 Nov 2011 10:48:44 GMT',
  'content-type': 'application/json; charset=utf-8'},
'{"access_token":"NEW_ACCESS_TOKEN","token_type":"bearer"}')


That's it. We now have our access token and are able to use it.

Get my user information using the access token:

>>> headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer NEW_ACCESS_TOKEN'}
>>> ret = http.request('https://api.flattr.com/rest/v2/user', headers=headers)
>>> ret
({'status': '200',
  'x-ratelimit-remaining': '4990',
  'content-location': 'https://api.flattr.com/rest/v2/user',
  '-content-encoding': 'gzip',
  'strict-transport-security': 'max-age=500',
  'vary': 'Accept-Encoding',
  'content-length': '274',
  'server': 'lighttpd',
  'connection': 'close',
  'x-ratelimit-limit': '5000',
  'date': 'Sun, 20 Nov 2011 10:54:01 GMT',
  'content-type': 'application/json'}, 
'{"type":"user","resource":"https:\\/\\/api.flattr.com\\/rest\\/v2\\/users\\/chrigl","link":"https:\\/\\/flattr.com\\/profile\\/chrigl","username":"chrigl","firstname":"Christoph","lastname":"Glaubitz","city":"","zip":"","province":"","cellphone":"","avatar":"","about":"","country":0}')
>>> simplejson.loads(ret[1])
{'about': '',
 'avatar': '',
 'cellphone': '',
 'city': '',
 'country': 0,
 'firstname': 'Christoph',
 'lastname': 'Glaubitz',
 'link': 'https://flattr.com/profile/chrigl',
 'province': '',
 'resource': 'https://api.flattr.com/rest/v2/users/chrigl',
 'type': 'user',
 'username': 'chrigl',
 'zip': ''}

And that's it. Thanks for reading, and don't forget to consult the Flattr API v2 documentation .

Autor
Chris Glaubitz
Configuring, coding, debugging computers for a living. Riding bikes for fun.

My Plone configuration

Basically this page is running on a - more or less - plain Plone 4 (currently 4.1.2 ). I just include some packages out of the collective.geo.*(search on pypi ) world, as well as my own plonetheme.sunburst based skin plonetheme.chrigl .

Because this is a really small page, I am only running a single Plone instance and there' s no need for a complex multi-instance-zeo-setup . Because there' s an already running apache2 on my server, I run my Plone behind it, using Zope's VirtualHostMonster .

Because my apache2 also serves home directories (e.g. http://chrigl.de/~chris/ ), my rewritings look like this:

RewriteCond %{REQUEST_URI} !^/~.*
RewriteCond %{REQUEST_URI} !^/icons/.*
RewriteCond %{REQUEST_URI} !^/build/.*
RewriteCond %{REQUEST_URI} !^/default-site/.*
RewriteCond %{REQUEST_URI} !^/error/.*
RewriteRule ^/(.*) http://localhost:8080/VirtualHostBase/http/%{SERVER_NAME}:80/Plone/VirtualHostRoot/$1 [L,P]

Since ^/~.*is user name, the other directories belong to apache2. It's necessary to pass these directories too, to find the folder, etc. images in apaches directory overview, as well as correct error handling.

Just for fun and testing, I'm running a varnish-3.0.1 cache in front of my instance (apache2 -> varnish -> Plone). With a proper configuration of varnish, it's possible to shut down the Plone instance, but still serve the page. However, at least as long as varnish cache the content. This is my varnish default.vcl

Plone is running on an Ubuntu 9.10 with upstart . So I decided to write two upstart scripts, one for the Plone instance and one for varnish.

/etc/init/plone.conf

# Plone - Web-Content Management System
#
# Based on Python and ZOPE

description "start chrigl's plone"
author "Christoph Glaubitz"
version "0.2"

console none
respawn
emits plone

start on (local-filesystems and net-device-up and runlevel [2345])
stop on runlevel [!2345]

exec /home/myuser/bin/plone-starter /home/myuser/Instances/chrigl/bin/instance console

[UPDATE]: Changed console to none, because of curios behavior of upstart.

Changed exec. Included plone-starter to emit the event plone. This triggers the start of plone-varnish. [/UPDATE]

/etc/init/plone-varnish.conf

# Varnish Cache
#
# ...is an open source, state of the art web application accelerator.
# You install it on your web server and it makes your website fly.

description "start varnish for chrigl's plone"
author "Christoph Glaubitz"
version "0.2"

console none
respawn

start on plone
stop on runlevel [!2345]

# run varnish as user myuser
exec su -s /bin/sh -c 'exec "$0" "$@"' myuser -- /opt/varnish-3.0.2/sbin/varnishd -F -f /home/myuser/Instances/chrigl/varnish/default.vcl -s malloc,100M -T 127.0.0.1:2000 -a 0.0.0.0:8080 -n /home/chris/Instances/chrigl/var/varnish

[UPDATE]: Changed console to none because varnish was not able to bind to ports when using "console output"... do not know why!?

Changed the exec string: appended -n /home/chris/Instances/chrigl/var/varnish, to give varnish a working directory else than the global /opt/varnish-3.0.2/var

Changed to start on plone. The script plone-tester emits this upstart event. However, now varnish is just started when plone is running. No bad idea I think. [/UPDATE]

Because varnish does not have something like Plone's effective-user, it is a little bit tricky to start it as non-root. Also this version of upstart does not provide a solution for this problem. However, my solution is borrowed from here (thanks guys!).

With this scripts, it's very easy to control the daemons via initctl, e.g.:

# initctl start plone
# initctl start plone-varnish
# initctl status plone-varnish
plone-varnish start/running, process 19046

Update: I decided to install varnish not directly into my buildout, but system-wide. Also the default.vcl is handmade and resists outside buildout.

Files:

Autor
Chris Glaubitz
Configuring, coding, debugging computers for a living. Riding bikes for fun.

Fiddled around with python-oauth2 and flattr

[UPDATE] This entry is not really valid any more, since flattr is switching to a rewritten api v2 . I will write a few words about v2 when I know how it works... And so I did :) [/UPDATE]

 

Because I like the idea of Flattr , I just fiddled around a little bit with python-oauth2 and Flattr's API .

This are the first steps of allowing your webapp to create a new thing on Flattr . This is very useful if you are running a podcast or blogging software, and you want to create a thing for each new entry or episode. So it's possible to flattr each of them separately, instead of the entire podcast/blog. E.g. there is a flattr plugin for Wordpress .

Remember this code is far away from being ready to use.

There are libraries for PHP , Ruby , as well as a community library for Java .

My entry point was the developer wiki of Flattr and the python-oauth2 documentation . It's also useful if you know the basics about oauth .

 

First of all, you need an account on Flattr , and register your webapp to retrieve the customer_keyand customer_secret. You have to choose browseras application type. At this point, you already have to know a callback_url.

You also have to install python-oauth2 into your python environment.

So let's play:

Let's create the customer:

>>> key='YOUR_CUSTOMER_KEY'

>>> secret='YOUR_CUSTOMER_SECRET'
>>> consumer = oauth.Consumer(key, secret)
>>> method = oauth.SignatureMethod_HMAC_SHA1()

Get the request token:

>>> params = {'oauth_consumer_key': key,

... 'oauth_timestamp': oauth.Request.make_timestamp(),
... 'oauth_nonce': oauth.Request.make_nonce(),
... 'oauth_version': oauth.Request.version,
... 'oauth_callback': 'YOUR_REGISTERED_CALLBACK_URL'}
>>> req = oauth.Request(url='https://api.flattr.com/oauth/request_token',
... parameters=params)
>>> req.sign_request(method, consumer, None)
>>> http = httplib2.Http(disable_ssl_certificate_validation=True)
>>> ret = http.request(req.to_url())
>>> ret
({'-content-encoding': 'gzip',
 'cache-control': 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0',
 'connection': 'close',
 'content-length': '158',
 'content-location': u'https://api.flattr.com/oauth/request_token?oauth_body_hash=SOME_HASH&oauth_nonce=75294181&oauth_timestamp=1318517637&oauth_consumer_key=YOUR_KEY&oauth_signature_method=HMAC-SHA1&oauth_version=1.0&oauth_signature=SIG&oauth_callback=http%3A%2F%2FYOUR_CALLBACK_URL',
 'content-type': 'application/x-www-form-urlencoded',
 'date': 'Thu, 13 Oct 2011 14:54:22 GMT',
 'expires': 'Thu, 19 Nov 1981 08:52:00 GMT',
 'pragma': 'no-cache',
 'server': 'lighttpd',
 'set-cookie': 'PHPSESSID=GENERATED_SESSION_ID; path=/; domain=.flattr.com; HttpOnly',
 'status': '200',
 'strict-transport-security': 'max-age=500',
 'vary': 'Accept-Encoding'},
 'oauth_token=NEW_REQUEST_TOKEN&oauth_token_secret=NEW_REQUEST_TOKEN_SECRET&oauth_callback=http://YOUR_CALLBACK_URL&oauth_callback_confirmed=true')

Unfortunately, in this code we are not able to use buildin clientof python-oauth2, because it is not possible to pass the callback_urlto client. However, we are able to fetch the request token with oauth2's Requestand httplib2.

In a real application, you have to store the request token and the secret permanently (of course, you only need it to get the access token).

>>> parse_ret = parse_qs(ret[1])

>>> request_token = parse_ret['oauth_token'][0]
>>> request_token_secret = parse_ret['oauth_token_secret'][0]

Once you have the request token, you must send the user (yourself) to Flattr, to authorize your webapp. Here, you only request permissions for readand publish. However, have a look at https://developers.flattr.net/doku.php/user_authentication for further information:

>>> 'https://api.flattr.com/oauth/authenticate?access_scope=read,publish&%s' % ret[1]

'https://api.flattr.com/oauth/authenticate?access_scope=read,publish&oauth_token=NEW_REQUEST_TOKEN&oauth_token_secret=NEW_REQUEST_TOKEN_SECRET&oauth_callback=http://YOUR_CALLBACK_URL&oauth_callback_confirmed=true'

The user (yourself) permits the webapp, and will be redirected to the configured callback_url. After that, you will find the oauth_verifierin the redirected url.

e.g. if you configured http://example.com/flattras callback_url, you will be redirected to http://example.com/flattr?oauth_token=NEW_REQUEST_TOKEN&oauth_verifier=VERIFIER_NUMBER

With your request token and the obtained oauth_verifier, you can fetch the access token. From this point python-oauth2 works fine, and we do not need to work around.

Create a token with the oauth_verifier:

>>> oauth_verifier=843099

>>> token = oauth.Token(request_token, request_token_secret)
>>> token.set_verifier(oauth_verifier)

Now you can use your consumerand the new tokento create a python-oauth2 client:

>>> client = oauth.Client(consumer, token)

The creation and signing of the request is now done by client. You do not have take care on it. You just have to pass the correct api url:

>>> ret = client.request('http://api.flattr.com/oauth/access_token', 'POST')

>>> ret
({'-content-encoding': 'gzip',
 'cache-control': 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0',
 'connection': 'close',
 'content-length': '96',
 'content-type': 'application/x-www-form-urlencoded',
 'date': 'Thu, 13 Oct 2011 14:55:38 GMT',
 'expires': 'Thu, 19 Nov 1981 08:52:00 GMT',
 'pragma': 'no-cache',
 'server': 'lighttpd',
 'set-cookie': 'PHPSESSID=i6md2vqg8t84hri1hk4r7448c3; path=/; domain=.flattr.com; HttpOnly',
 'status': '200',
 'vary': 'Accept-Encoding'},
 'oauth_token=NEW_ACCESS_TOKEN&oauth_token_secret=NEW_ACCESS_TOKEN_SECRET')
>>> parse_ret = parse_qs(ret[1])
>>> access_token = parse_ret['oauth_token'][0]
>>> access_token_secret = parse_ret['oauth_token_secret'][0]

You have to store the access token and the secret permanently. It is valid, till the user (yourself) rejects the permission on Flattr (Settings -> Tab Application). Now you can delete the request token and secret (request not the access token!!!)

Your application is registered now, and you granted the necessary permissions. In this case readand publish. Use the consumer and the access token to speak with Flattr API .

E.g. request your user information. It is again very easy to use python-oauth2 without work arounds:

>>> token = oauth.Token(access_token, access_token_secret)

>>> client = oauth.Client(consumer, token)
>>> ret = client.request('http://api.flattr.com/rest/0.0.1/user/me', 'GET')
>>> ret                                                                         
({'-content-encoding': 'gzip',
 'connection': 'close',
 'content-length': '276',
 'content-location': u'http://api.flattr.com/rest/0.0.1/user/me?oauth_body_hash=SOME_HASH&oauth_nonce=55948733&oauth_timestamp=1318518733&oauth_consumer_key=APPS_KEY&oauth_signature_method=HMAC-SHA1&oauth_version=1.0&oauth_token=YOUR_ACCESS_TOKEN&oauth_signature=SOME_SIG',
 'content-type': 'text/xml',
 'date': 'Thu, 13 Oct 2011 15:12:13 GMT',
 'server': 'lighttpd',
 'status': '200',
 'vary': 'Accept-Encoding'},
 '<?xml version="1.0" encoding="utf-8"?>\n
  <flattr>
   <version>0.0.1</version>
   <user><id>YOUR ID</id>
    <username>YOUR USERNAME</username>
    <firstname>YOUR NAME</firstname>
    <lastname>YOUR LAST NAME</lastname>
    <description></description>
    <thingcount>2</thingcount>
    <language>en_GB</language>
   </user>
  </flattr>\n')

For a ... hmm, not really complete ... documentation of what you can do, have a look at the Flattr API documentation .

Hoping to give some inspiration.

Autor
Chris Glaubitz
Configuring, coding, debugging computers for a living. Riding bikes for fun.

GPS Track meiner Skandinavientour ist online

... wenn auch ein bisschen verspätet habe ich den GPS Track meiner Skandinavientour endlich online. Sogar inklusive navigierbarer Karte, die mittels OpenLayers eingebunden ist.

Während der Tour habe ich täglich eine Datei gespeichert, damit einzelne Dateien nicht zu groß wurden und irgendwelche unerwarteten Probleme erzeugen. Natürlich konnte ich die Daten aus dem GPS nicht direkt nehmen und hochladen. Die Masse an gesammelten Punkten war einfach zu groß und haben damit zu lange Ladezeiten zur Folge hatten. Dank GPSBabel habe ich die täglichen Dateien zu einer kombiniert und überflüssige Punkte entfernt.

  • Track
Autor
Chris Glaubitz
Configuring, coding, debugging computers for a living. Riding bikes for fun.

Neue Seite, neues Layout

Habe mich dazu entschieden endlich mal wieder eine halbwegs schöne Seite an den Start zu bringen. Außerdem war ich es leid, dauernd per Hand in HTML-Dateien zu basteln und da ich in den letzten Jahren extrem viel mit Plone gemacht habe, setzt meine Seite jetzt auch darauf auf.

 

Heute habe ich ein Update auf auf Version 4.0.7 gemacht, auch wenn das nocht nicht als stabile Version rausgegeben wurde. Damit habe ich mir auf jeden Fall das gesonderte Einspielen des letzten Hotfixes gespart.

Autor
Chris Glaubitz
Configuring, coding, debugging computers for a living. Riding bikes for fun.