Thursday, February 28, 2013
Neues Blog zur funktionalen Programmierung
Wir haben ein neues Blog!
Zusammen mit der active group GmbH zeigen wir im neuen Blog jede Woche an Beispielen Vorteile und Einsatzgebiete der funktionalen Programmierung auf. Wir möchte über diese neue Platform das Wissen teilen, das wir im langjährigen, kommerziellen Einsatz dieser Techniken erworben haben. Wir hoffen, dass unsere Erfolge und guten Erfahrungen helfen können auch andere deutsche Softwareentwickler und Firmen zu ermutigen funktionale Programmierung einzusetzen.
Trotz des neuen Blogs werden wir weiterhin auf dem Firmenblog hier interessante technische Aspekte und Firmenneuigkeiten mit unseren Lesern teilen. Artikel über funktionale Programmierung werden aber schwerpunktmäßig im neuen Blog zu finden sein. Es lohnt sich also in Zukunft beide Blogs zu abbonieren.
Thursday, December 27, 2012
Templated PDF Printing for Appcelerator based iOS Apps
Content
- Motivation
- Idea of a solution
- Detailed explanation
- IOS Module
- HTML-Template
- Inline-Images
- HTML Page-Formatting
Motivation
With Javascript being the language of choice within the appcelerator framework,a huge variety of libraries and techniques for common problems, the catch being,
that many of these depend on the context of a web browser to funtion properly.
When it comes to creating PDF data within Appcelerator apps, so far the approach
had therefore been to small pure JS libs, such as jsPDF (http://jspdf.com/).
Yet, because of it's lowlevel approach (of drawing rectangles and lines) and
rendering text, it can be cumbersome to create a fully layouted document.
Idea of a solution
As an alternative, I propose to make use of Appcelerators module framework, i.e.,making use of native code, to pass an html string to an IOS module that will
render the content in a UIWebview object and subsequently to a PDF file, that
is returned to the application layer.
IOS Module
The module used for this solution has a method "setHTMLString", that can becalled from the application layer with the HTML string to be rendered in
a UIWebview object. Upon loading, the method "webViewDidFinishLoad" is called,
where the page then is rendered to a PDF file, subsequently firing an event
"pdfready", including a path to the file. This is necessary, as events in
Appcelerator can only carry JSON serializable data. Because the UIWebkit library
is not thread safe, all the calls in "setHTMLString" and "webViewDidFinishLoad"
have to be queued on the main thread by using "dispatch_async". For details check
the source code from github, referenced below. For general information about
developing iOS modules, check https://wiki.appcelerator.org/display/guides/iOS+Module+Development+Guide
HTML Template
Since we are passing an HTML string from the application layer and basicallyrender it in a browser window, all known HTML traits apply and the document
can be layouted, as applicable. But, since there usually will be no webserver
running to pull resources from, you have to inline any CSS or JS code into the
HTML string, you pass to the IOS module. Based on such a template you might
create your final HTML string by replacing certain placeholders, such as
<!--PLACEHOLDER--> with content data. Or you choose to just construct your
HTML string, as needed. Consider the following Appcelerator code:
var html2pdf = require('com.factisresearch.html2pdf');
Ti.API.info("module is => " + html2pdf);
html2pdf.addEventListener('pdfready', function(e) {
var emailDialog = Ti.UI.createEmailDialog();
emailDialog.addAttachment(Ti.Filesystem.getFile(e.pdf));
emailDialog.open();
});
var html = '<html><body><p>Hello World!</p></body></html>';
html2pdf.setHtmlString(html);
Inline Images
Of course it should be possible to add images to the document. Because generallyit will not be possible to pull resources, the images also have to be embedded
into the HTML string. How this can be done is explained here:
http://www.techerator.com/2011/12/how-to-embed-images-directly-into-your-html/
and in countless other places. Consider the following code:
var html2pdf = require('com.factisresearch.html2pdf');
Ti.API.info("module is => " + html2pdf);
html2pdf.addEventListener('pdfready', function(e) {
var emailDialog = Ti.UI.createEmailDialog();
emailDialog.addAttachment(Ti.Filesystem.getFile(e.pdf));
emailDialog.open();
});
var html = '<html><body>';
var imageData = Titanium.Filesystem.getFile(Ti.Filesystem.getResourcesDirectory(), 'Default.png').read();
var iv = Ti.UI.createImageView({
image: imageData
});
var image = iv.toImage();
var imageB64 = Ti.Utils.base64encode(image);
html += '<p><img src="data:image/png;base64,'+imageB64+'"></p></body></html>';
html2pdf.setHtmlString(html);
HTML Page-Formatting
The IOS module described above uses static page sizes and bounds for the printablearea. You may however want to customize this for different pages. On this account
CSS offers a range of customization options, such as the "@page" attribute, where
margins can be defined or the "page-break-before" style, that can set on an html
element to enforce the subsequent content to be displayed on a new page in the
resulting pdf.
-------------------------------
var html2pdf = require('com.factisresearch.html2pdf');
Ti.API.info("module is => " + html2pdf);
html2pdf.addEventListener('pdfready', function(e) {
var emailDialog = Ti.UI.createEmailDialog();
emailDialog.addAttachment(Ti.Filesystem.getFile(e.pdf));
emailDialog.open();
});
var html = '<html><body>';
var works = queryDBForWorks();
for (var i in works) {
var entry = {};
var rs = db.getWorkInfo(works[i]);
html += ('<h2 '+(i == 0 ? '>' : 'style="page-break-before:always;">')+''+rs.fieldByName('TITLE')+' - '+rs.fieldByName('ARTIST')+'</h2>');
html += ('<img src="data:image/jpg;base64,'
+Ti.Utils.base64encode(rs.fieldByName('THUMBNAIL'))+'">');
html += ('<p>'+''+rs.fieldByName('TEXT')+'</p></div>');
rs.close();
}
html2pdf.setHtmlString(html);
You may pull an implementation, along with an example from
https://github.com/fscz/html2pdf
Sunday, October 7, 2012
Developing Medical Software in Scala and Haskell
By the way, CUFP hosted a whole bunch of very interesting talks. They are all available on youtube, just search for "CUFP 2012".
Thursday, August 9, 2012
RedmineR - Redmine iOS App
Die Verwendung der App ist denkbar einfach: Zuerst muss "Redminer" kostenlos aus dem App Store geladen werden. Danach wird der API-Zugriffschlüssel benötigt - dieser wird in der normalen Redmine-Oberfläche unter "Mein Konto > API-Zugriffsschlüssel" angezeigt. Der Zugriffschlüssel wird dann auf dem iOS-Gerät unter "Einstellungen > Redminer" bei "API-Key" eingetragen. Außerdem muss hier noch die Tracker-URL eingetragen werden.
Download und weitere Informationen: Redminer für iPhone
Screenshots:
Tuesday, July 24, 2012
Android-App: Anrufweiterleitung
Sunday, June 3, 2012
UIs for Hierachical iPad Apps
If you're in a hurry: tl;dr version
Full-blown post
Motivation
How does it work?
 |
| regular view |
 |
| after revealing more information in the level-2 view controller (the "Teams" view controller) by swiping to the right |
Can I use FRLayeredViewController, too?
What is the API like?
Say, you have a UIViewController and you're using UINavigationController from Apple. To show a new layer to the user, you would do:
[self.navigationController pushViewController:test animated:YES];
[self.layeredNavigationController pushViewController:test
inFrontOf:self
maximumWidth:NO
animated:YES];
The maximumWidth parameter is how you tell FRLayeredNavigationController whether the new view controller is content (give it the maximal width still available) or not.
Just as UINavigationController, it also supports UIBarItems and a configurable title view. For example, a UIViewController which has been pushed onto a FRLayeredNavigationController could have the following viewWillAppear method to add a back button on the upper left corner:
- (void)viewWillAppear:(BOOL)animated
{
self.layeredNavigationItem.leftBarButtonItem =
[[UIBarButtonItem alloc]
initWithImage:[UIImage imageNamed:@"back-button.png"]
style:UIBarButtonItemStylePlain
target:self
action:@selector(backButtonClicked)];
self.layeredNavigationItem.leftBarButtonItem.style =
UIBarButtonItemStyleBordered;
}
How can I embed FRLayeredViewController in my project?
Thursday, December 1, 2011
A rant about Scala vs. Haskell
Scala's type system is powerful but also complex right from the start. With Haskell you can use most basic libraries without enabling GADTs, RankNTypes, OverlappingInstances and TypeFamilies or having to learn about them. And you can take your time to learn all these concepts. The containers API is really simple compared to Scala's collections API. On the other hand in Haskell you have to learn about monads quite early. (Oh, I miss this warm fuzzy feeling in Scala!)
We're using Scala for integration with existing Java code and libraries and that's working really well. As a "stand alone" language I still prefer Haskell and I doubt that's going to change with more Scala experience. For me as a functional programmer subtyping doesn't add much more than Java compatibilty. Maybe I have to change but IMHO it makes API design and the type system too complex. I'm still overwhelmed by the "default" design possibilities: type parameters or abstract member types? Sealed case classes or open subtyping? Implicit parameters and conversions (aka type classes) offer even more design choices! Maybe I'm just more experienced with Haskell but mostly I just decide between "data" and "class" and with higher-order functions that's enough 95% of the time. For the missing 5% we have extensions (and FlexibleX + Rank2Types + TypeFamilies + ExistentialQuantification is enough for me).
Often people complain about GHC's error messages but they're really great compared to scalac's. I also miss STM. Actors are nice, but using react doesn't feel natural and we can't just use "receive" with thousands of threads "without thinking" just like in Haskell. (We're now using react with the CPS transform plugin to hide the fact that internally everything is implemented using continuations.)
I don't miss lazyness, yet. I've used call-by-name and lazy vals on some occasions and I think I prefer strict by default with optional lazyness annotations. I just can't think "the lazy way" - even after 18 months of full-time Haskell coding.
I do miss control over side effects. When writing a higher order function I always get scared when I realize that I can only expect the caller to provide a pure function but the function could be doing all sorts of things behind my back!
One thing I'd like as a GHC extensions is explicit dictionary passing. It's nice that implicit parameters can be used implicitly but also explicitly. Of course it's risky, too, because nobody can stop you from using a different Ord instance every time you're inserting into a Set. "unsafeCoerce" seemed so evil - until erasure forced me to use foo.asInstanceOf[...] (Of course an exception is not as bad as a segfault.) Sometimes I know what I want and I'm willing to take the risk. As long as all those features are not enabled by default I think hard enough before enabling {-# LANGUAGE IncoherentInstances #-}.
Scala might be the best way to live between worlds. But you have to understand both worlds and you're still in between. It's not twice as good. If I have to live in OO world I prefer having a functional world in addition so I'd chose Scala over Java at any time. But if I can choose freely I still prefer the pure Haskell world.
Sunday, October 30, 2011
WifiAlive Android App released
WifiAlive is a very simple and low resource application aimed at maintenance of a Wifi connection. A connection is periodically (every 5 seconds) checked by trying to contact the gateway, as well as IP addresses or domain names which you can enter via the UI. WifiAlive tests each of these addresses by icmp and http head. If one of these tests succeeds, Wifi is considered up. If not, Wifi is reset (that is deactivated and activated). After Wifi (re-)activation Android on its own tries to establish a connection to an access point in reach. On a failed connection check, WifiAlive will only wait 20 seconds for a reset, scan and successful connection until the connection is reset again. Considering the time needed for a Wifi reset and a scan to complete, it can be said, that, as soon as a known network is available, there is a worst-case of roughly 30 seconds for the connection to be established. WifiAlive is thus much more simple then many of its competitors. The popular WifiFixer for example conveys about 6000 lines of code whereas WifiAlive has only around 500.
The code of WifiFixer is publicly available on github. I have checked those parts of the code which seemed important in respect of a problem I believed to have with WifiFixer 0.8.0.6. I was rather stunned by the complexity of the program, since I had in mind the solution I have just proposed. WifiFixer almost replaces the Wifi Settings Tab in Android and will continue this approach in upcoming versions. In some cases, settings from the Android Wifi Settings Tab can be overridden, it does its own management of known networks and behind the scenes does scans for networks in reach, matches them to known networks and harbours a small collection of hacks for devices that have to be dealt with in a special way - for example the Google Nexus One. WifiFixer is a great application. It helps many people with their Wifi troubles. But after I had seen its complexity and, in addition, found a bug in its matching of known to found networks on a scan, I decided that this was not the solution to my problem, as, additionally, a lot of code seemed to do things that Android normally does on its own. No offence David, I've learned a lot from looking at your code.
There are other applications, like Advanced Wifi Lock. But I couldn't get access to their code and very often the free (limited) versions did not seem to cover all aspects of Wifi maintenance that I wanted to ensure. Moreover I am already beyond trusting a random application from the market without knowing its code. The Advanced Wifi Lock application (a bare Wifi lock, if it does what the name suggests) does not solve my problems, as it sometimes takes Android a bit too much of time to re-establish a connection when re-entering the area of an access point or on walking from one access point to another. In my experience this could take up to two minutes, which was just too long for my purpose. These experiences may be better on other devices, still it feels very good to know, that as soon as a known network is in reach, there is a rough 30 second worst-case until you have a connection.
On some devices there also seem to be problems with the supplied Wifi drivers. These drivers may enter an undetermined state or cause wpa supplicant to run in endless loops. I have not delved into these problems. But it seemed to me, that, if at all, they can be resolved by doing a Wifi reset.
To sum up, I think that some applications aimed at taking care of your Wifi connection may be too complicated and question is, why such an application should do things like scanning for access points, as Android usually does a pretty good at this. Others may be doing the things I had in mind, yet there was no way for me to be sure.
That's, why I wrote WifiAlive. I hope you like it.
Feel Free to browse the code at http://darcs.factisresearch.com/pub/WifiAlive or just install the app from the android market.
Sunday, October 9, 2011
stm-stats: Retry statistics for STM transaction
import Control.Concurrent import Control.Concurrent.STM import Control.Monad import Control.Concurrent.STM.Stats main = do var <- trackSTM $ newTVar 0 forkIO $ forM_ [1..23] $ \i -> do threadDelay (100*1000) trackNamedSTM "writer" $ writeTVar var i putStrLn "Starting reader..." trackNamedSTM "reader" $ do i <- readTVar var when (i < 23) retry putStrLn "Reader finished." dumpSTMStatsWhen run, you will see this output:
Starting reader... STM transaction reader finished after 23 retries Reader finished. STM transaction statistics (2011-10-09 16:26:27.226675 UTC): Transaction Commits Retries Ratio _anonymous_ 1 0 0.00 reader 1 23 23.00 writer 23 0 0.00PS: As I usually post on my own blog, I should explain why I from now on will also post on the factis research company blog. Factis research relies heavily on Free Software and wants to contribute back to the community where possible. But in the course of every-day work, this sometimes falls by the wayside. Therefore, I was hired to help out as the community interface: My task is identifying, packaging and uploading components of internal code that are of wider interest, following up on user requests and bug reports, and talk about it. This module is the first brought to you by this new strategy, but expect more to come.
Saturday, October 8, 2011
Talk about developing commercial software in Haskell
Yesterday, I gave a talk at the Haskell in Leizpig workshop about our experience in developing commercial software in Haskell. You can find the slides (in german) here. I was really surprised by the large number of attendees (about 50), given that the workshop's language was german (except the very interesting talk by Kevin Hammond).
Saturday, October 1, 2011
New Version of HTF with Diffs, Colors, and Pretty-printing
I've just uploaded version 0.8.1.0 of HTF to hackage. HTF (Haskell Test Framework) allows you to define unit tests, QuickCheck properties, and black box tests in an easy and convenient way. We use HTF at work all the time, where it has proven to be quite valuable for organizing our test suite. An earlier blog post describes HTF in more detail.
The new version comes with some cool new features:
-
Support for diffs and pretty-printing.
If an equality assertions fails, you now get a proper diff of
the two values involved. If possible, the values are pretty-printed
(thanks to Edward Yang and his groom
package for inspiration). Here's an example:
[TEST] Main:diff (TestHTF.hs:68) assertEqual failed at TestHTF.hs:68 * expected: PlusExpr (PlusExpr (MultExpr (PlusExpr (Variable "foo") (MultExpr (Literal 42) (Variable "bar"))) (PlusExpr (Literal 1) (Literal 2))) (Literal 581)) (Variable "egg") * but got: PlusExpr (PlusExpr (MultExpr (PlusExpr (Variable "foo") (MultExpr (Literal 42) (Variable "bar"))) (PlusExpr (Literal 2) (Literal 2))) (Literal 581)) (Variable "egg") * diff: 6c6 < (PlusExpr (Literal 1) (Literal 2))) --- > (PlusExpr (Literal 2) (Literal 2))) *** Failed! - As you can see from the example above, HTF now supports colored output.
- There's a new commandline option --quiet which causes HTF to produce output only if absolutely necessary (e.g. for failed test cases).
Just get HTF from hackage, it now also works with GHC 7.0.* and 7.2.1!
Wednesday, May 18, 2011
xmlgen: a feature-rich and high-performance XML generation library
I’ve released xmlgen to hackage just a few days ago. Xmlgen is a pure Haskell library with a convenient API for generating XML documents. It provides support for all functionality defined by the XML information set and offers good performance and low memory usage. In our company, we developed xmlgen because we wanted the readability of XML literals (as for example provided by the hsp library) without the drawbacks of a custom preprocessor (wrong line numbers in error messages, non-compositionality).
In this blog article, I’ll show you how to use the combinators provided by xmlgen to generate the following XML document:
<?xml version="1.0"?> <people> <person age="32">Stefan</person> <person age="4">Judith</person> </people>
First, we import some modules:
> import Data.Monoid > import qualified Data.ByteString.Lazy as BSL > import Text.XML.Generator -- the main xmlgen module
Then we continue by generating the person element.
> genPersonElem :: (String, String) -> Xml Elem > genPersonElem (name, age) = > xelem "person" $ xattr "age" age <#> xtext name
The xelem combinator constructs an XML element from an element name and from the children of the element. Xmlgen provides overloaded variants of xelem to support a uniform syntax for the construction of elements with qualified and unqualified names and with different kinds of children. The <#> combinator separates the element’s attributes from the other children (sub-elements and text nodes). The combinators xattr and xtext construct XML attributes and text nodes, respectively.
The result of an application of xelem has type Xml Elem, whereas xattr has result type Xml Attr. This distinction is important so that attributes and elements can not be confused. The result type of the xtext combinator is Xml Elem; we decided against an extra type for text nodes because for xmlgen’s purpose text nodes and elements are almost interchangeble.
The types Xml Elem and Xml Attr are both instances of the Monoid type class. Constructing a list of elements from a list of persons and their ages is thus quite easy:
> genPersonElems :: [(String, String)] -> Xml Elem > genPersonElems = foldr mappend mempty . map genPersonElem
The pattern shown above is quite common, so xmlgen allows the following shorthand notation using the xelems combinator.
> genPersonElems' :: [(String, String)] -> Xml Elem > genPersonElems' = xelems . map genPersonElem
We are now ready to construct the final XML document:
> genXml :: Xml Doc
> genXml = let people = [("Stefan", "32"), ("Judith", "4")]
> in doc defaultDocInfo $ xelem "people" (genPersonElems people)
For convenience, here is a standalone variant of the genXml function:
> genXml' :: Xml Doc
> genXml' =
> let people = [("Stefan", "32"), ("Judith", "4")]
> in doc defaultDocInfo $
> xelem "people" $
> xelems $ map (\(name, age) -> xelem "person" (xattr "age" age <#> xtext name)) people
Xmlgen supports various output formats through the overloaded xrender function. Here we render to resulting XML document as a lazy byte string:
> outputXml :: IO () > outputXml = BSL.putStrLn (xrender genXml')
Loading the current file into ghci and evaluating outputXml produces the following result:
*Main> outputXml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <people ><person age="32" >Stefan</person ><person age="4" >Judith</person ></people >
This blog post introduced most but not all features of the xmlgen API. Check out the documentation.
Happy hacking and have fun!
Author: Stefan Wehr
Friday, April 29, 2011
Empty GHC profile files and signal handling
The GHC Commentay http://hackage.haskell.org/trac/ghc/wiki/Commentary/Rts/Signals explains that the default interrupt handler calls ”interruptStgRts” to exit the program. We now import this C-function via FFI and call it in our signal handler.
Update! We lose the ability to specify an exit code when using “interruptStgRts” so we now call “shutdownHaskellAndExit” instead.
Here’s an example program that demonstrates the use of System.Posix.Process and the FFI call to “shutdownHaskellAndExit”:
{-# LANGUAGE ForeignFunctionInterface #-}
import Control.Concurrent
import Foreign.C( CInt )
import System.IO
import System.Posix.Process
import System.Exit
import System.Posix.Signals
foreign import ccall "shutdownHaskellAndExit" shutdownHaskellAndExit :: CInt -> IO ()
firstSigINT :: IO ()
firstSigINT =
do hPutStrLn stderr "caught interrupt (shutdown takes 2s)"
hFlush stderr
installHandler keyboardSignal (Catch secondSigINT) (Just emptySignalSet)
threadDelay (2*1000*1000)
shutdownHaskellAndExit 13
secondSigINT :: IO ()
secondSigINT =
do hPutStrLn stderr "forcing immediate exit"
hFlush stderr
exitImmediately (ExitFailure 13)
main =
do installHandler keyboardSignal (Catch firstSigINT) (Just emptySignalSet)
let busyloop i = busyloop (i+1)
busyloop 0
Author: David Leuschner
Thursday, April 14, 2011
Re-Signing von iOs-Apps
Mit folgenden, einfachen Schritten ist ein Signieren einer bestehenden ipa-Datei möglich:
- in ein Temp-Verzeichnis wechseln:
cd tmp - ipa-Datei auspacken:
unzip.ipa
Ergebnis: Unterordner "Payload", der einen Ordner für die App mit dem Name "Appname.app" enthält
- AdHoc-Profil ersetzen:
cp.mobileprovision Payload/ .app/embedded.mobileprovision - Neu signieren:
codesign -f -vv -s "iPhone Distribution" Payload/.app - Packen:
zip -r.ipa Payload
Autor: Dirk Spöri
Thursday, February 17, 2011
Microbenchmark of Haskell MD5 implementations
On my Intel i7 860 the results are:
import qualified Data.ByteString as BS
import qualified Data.ByteString.Lazy as BSL
import qualified Data.Digest.OpenSSL.MD5 as NanoMD5
import qualified Data.Digest.Pure.MD5 as PureMD5
import qualified Crypto.Hash.MD5 as ChMD5
import Numeric (showHex)
import Criterion.Main (defaultMain, bench, nf)
import System.Environment (getArgs)
main =
do x <- BSL.readFile "/lib/libc.so.6"
BSL.length x `seq`
defaultMain [ bench "cryptohash" $ nf ch x
, bench "nano" $ nf nano x
, bench "pure" $ nf pure x
]
go :: BS.ByteString -> Int -> [String] -> String
go bs n acc
| n `seq` bs `seq` False = undefined
| n >= 16 = concat (reverse acc)
| otherwise = go bs (n+1) (draw (BS.index bs n) : acc)
draw w =
case showHex w [] of
[x] -> ['0', x]
x -> x
nano :: BSL.ByteString -> String
nano lazy =
let strict = BS.concat $ BSL.toChunks lazy
in NanoMD5.md5sum strict
pure :: BSL.ByteString -> String
pure = show . PureMD5.md5
ch :: BSL.ByteString -> String
ch bs = go (ChMD5.hashlazy bs) 0 []
benchmarking cryptohash
mean: 2.886409 ms, lb 2.885261 ms, ub 2.887946 ms, ci 0.950
std dev: 6.718781 us, lb 5.388123 us, ub 8.687483 us, ci 0.950
benchmarking nano
mean: 2.704862 ms, lb 2.704301 ms, ub 2.706016 ms, ci 0.950
std dev: 3.970617 us, lb 2.260051 us, ub 7.420531 us, ci 0.950
benchmarking pure
mean: 10.27061 ms, lb 10.26878 ms, ub 10.27485 ms, ci 0.950
std dev: 13.54268 us, lb 7.179537 us, ub 27.38285 us, ci 0.950
Author: David Leuschner
Saturday, November 13, 2010
Netboot Server with Gentoo and AUFS
Abstract
This Howto describes the installation of a gentoo server for a netboot system that uses aufs for user write layers, logical volume management (lvm), raid (mdadm) and ubuntu as guest os. I assume, you start with nothing at all and have to install the server os first.BOOT FROM CD
There are a number of different ways to install gentoo. Here we do it from scratch, as it will hopefully provide you with some understanding of what you are doing and how this system works. Fetch the systemrescuecd from the link, supplied below, burn it, put it into your drive and boot from it. You may also use your favorite installation disk, lest it includes lvm and mdadm. http://www.sysresccd.org/DownloadSETUP DISK(s)
Partitions
We create a software raid using mdadm. So assuming we have two real disks, we will create two partitions, each. The first partition will be very small and is only needed for the /boot folder. Grub supports only version 1.0 of a mdadm raid, thats why we use –metadata=1.0. Also we use raid1, because grub does not support raid 10. The second partition will comprise the rest of available disk space and can for example be of raid type 10, so at any time, one disk may fail and we can recover our data. A raid 10 with two disks, thus behaves like raid 1. On top of this raid 10, we set up the logical volume manager and use logical partitions for data, while being flexible with space distribution, in case we add disks in the future. The setup of the logical partitions below is only a proposition that has proven practical. You may want to create a different setup. But you will want at least one extra partition for the guest os./dev/sda:
-/dev/sda1, set boot flag, >= 200mb (this will be the boot partition
-/dev/sda2 = rest (this will be our gentoo server and client system)
/dev/sdb: create EXACTLY the same layout as for /dev/sda
Software Raid
create your raid devicesmdadm –create /dev/md0 –level=1 –raid-devices=2 /dev/sda1 /dev/sdb1 –metadata=1.0
mdadm –create /dev/md1 –level=10 –raid-devices=2 /dev/sda2 /dev/sdb2
LVM
create lvm for “server” and “client” diskvgcreate system /dev/md1
lvcreate -n server-root -L 20G system
lvcreate -n server-swap -L 4G system # this should be twice your ram size
lvcreate -n client-boot -L 200M system
lvcreate -n client-root -L 50G system # no need to save space here
lvcreate -n client-home -L 500G system # as much as you need
Filesystems
mkfs.ext2 /dev/md0mkfs.ext4 /dev/system/server-root
mkswap /dev/system/server-swap
mkfs.ext2 /dev/system/client-boot
mkfs.ext4 /dev/system/client-root
mkfs.ext4 /dev/system/client-home
Mount
mkdir /mnt/gentoomount /dev/system/server-root /mnt/gentoo
mkdir /mnt/gentoo/boot
mount /dev/md0 /mnt/gentoo/boot
mkdir /mnt/gentoo/dev
mkdir /mnt/gentoo/proc
mount -t proc none /mnt/gentoo/proc
GENTOO INSTALLATION
We set up a basic gentoo installation. Nothing special here. Just follow the instructions and you’ll be fine. If you’re not familiar with gentoo, you can get a pretty good idea from the official gentoo howtos at the link below.follow instructions from official handbook to obtain a stage and portage snapshot and unpack them http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=1&chap=5 cp -L /etc/resolv.conf /mnt/gentoo/etc/
mount –bind /dev /mnt/gentoo/dev
chroot /mnt/gentoo /bin/bash
env-update; source /etc/profile
cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime
nano /etc/locale.gen
en_US ISO-8859-1
en_US.UTF-8 UTF-8
de_DE ISO-8859-1
de_DE@euro ISO-8859-1
locale-gen
emerge –sync
INSTALL SERVER
Here we begin customization. Aside from standard tools, we install tftpd-hpa, which basically IS the netbootservice, mdadm and lvm. After the installation we add the services to default runlevel. We continue by creating directories, in which our guest operating system, ubuntu, will be installed and set up /etc/fstab accordingly. Feel free to choose different paths, if you like. Installation of nfs-utils should be clear. In /etc/hosts.allow we define, which machines are allowed to boot via network. You will probably have a different setup here. Use ip ranges according to your needs. /etc/hosts.deny is called only after hosts.allow so you might want to deny everything else. Then we set the hostname and path that tftpd will look for a kernel to boot via network. This is, where we will put the guest os /boot dir. Setup of mdadm follows. Here we specifiy, which disks go into which raid array. We use genkernel to create a kernel, ramdisk and modules for our purpose and then patch our kernel with aufs and set it to autoload. This is necessary, since aufs is not an official part of the kernel. After installing grub, we’re good to reboot.eselect profile set 7
passwd # set a secure server password, for example 7531
emerge -av =gentoo-sources-2.6.34-r1
emerge -av sysklogd vixie-cron ssmtp ntp eix htop dhcpc openssh tftp-hpa mdadm grub genkernel
ACCEPT_KEYWORDS=”~x86″ emerge -av =sys-fs/lvm2-2.02.72
rc-update add sysklogd default; rc-update add vixie-cron default; rc-update add sshd default; rc-update add ntpd default; rc-update add ntp-client default
nano /etc/conf.d/net
config_eth0=( “dhcp” )
mkdir -p /tftpboot/static/root
mkdir -p /tftpboot/static/home
mkdir -p /tftpboot/static/boot
nano /etc/fstab
/dev/md0 /boot ext2 defaults 0 0
/dev/system/server-root / ext4 defaults 0 1
/dev/system/server-swap none swap sw 0 0
/dev/system/client-boot /tftpboot/static/boot ext2 defaults 0 0
/dev/system/client-root /tftpboot/static/root ext4 defaults 0 0
/dev/system/client-home /tftpboot/static/home ext4 defaults 0 0
none /proc proc defaults 0 0
USE=”selinux nonfsv4 tcpd” emerge -av nfs-utils
rc-update add nfs default
nano /etc/hosts.allow
ALL: 10.11.0.0/16
ALL: 10.10.0.0/16
ALL: 10.20.0.0/16
nano /etc/hosts.deny
ALL: (ALL)ALL
nano /etc/conf.d/hostname
HOSTNAME=”moros”
rc-update add nfs default
nano /etc/conf.d/in.tftpd
INTFTPD_PATH=”/tftpboot/static/boot”
rc-update add in.tftpd default
nano /etc/mdadm.conf
DEVICE /dev/sda*
DEVICE /dev/sdb*
ARRAY /dev/md0 metadata=1.0 devices=/dev/sda1,/dev/sdb1
ARRAY /dev/md1 metadata=1.1 devices=/dev/sda2,/dev/sdb2
genkernel –install –menuconfig –lvm –mdadm all
ACCEPT_KEYWORDS=”~x86″ USE=”nfs kernel-patch” emerge aufs2
nano /etc/modules.autoload.d/kernel-2.6
aufs
nano /boot/grub/grub.conf
default 0
timeout 30
title Gentoo Linux 2.6.34-r1
root (hd0,0)
kernel /boot/kernel-genkernel-x86-2.6.34-gentoo-r1 root=/dev/ram0 real_root=/dev/system/server-root domdadm dolvm
initrd /boot/initramfs-genkernel-x86-2.6.34-gentoo-r1
grub #this might take some time (7 min)
Grub>device (hd0) /dev/sda (/dev/hda for ide)
Grub>root (hd0,0) and then:
Grub>setup (hd0) # this might take some time
Grub>device (hd1) /dev/sdb (/dev/hdb for ide)
Grub>root (hd1,0) and then:
Grub>setup (hd1) # this might take some time
quit #this might take some time
———————–
DHCP INSTALLATION / SETUP
Since booting from the network requires the client to send broadcasts and listen for a response from a dhcp server that, rough said, contains the kernel to boot, we need to either install a new dhcp server or add a few lines of code to our existing server. The critical lines here are “next-server”, this is the ip address of you netboot server and “filename”. Just leave the filename as displayed. You will understand shortly. There are some possibilities to tell a client, which root path to use, respectively which nfs to mount as /. Since we use aufs to give every client the chance to customize the system in his own way thus having a shared base, that can be configured individually, we use different root paths for each machine. Each of those root paths consists of a static layer that comprises all the shared data and a writeable layer, in which per client data will be saved. Creation of these aufs filesystem follows. Use the information below to find your best way to set up dhcp.emerge dhcp
nano /etc/conf.d/dhcp
INTERFACES=’”eth0″
nano /etc/dhcp3/dhcpd.conf
subnet 192.168.5.0 netmask 255.255.255.0 {
range 192.168.5.100 192.168.5.254;
option domain-name-servers 10.7.0.1;
option routers 192.168.1.253;
option broadcast-address 192.168.5.255;
default-lease-time 600;
max-lease-time 7200;
next-server 192.168.5.1;
##for each host
host 192.168.5.100 {
hardware ethernet 00:25:64:8e:16:c4;
fixed-address 192.168.5.100;
filename “pxelinux.0″;
option root-path “/tftpboot/dynamic/10.7.0.<ip>”; # this is perhaps the most sofisticated method to get your root fs mounted. see another possibility below }
————————
SETUP PXELINUX
Think of pxelinux as some kind of network-grub. We emerge syslinux but are interested only in one file: pxelinux.0. This is the file, you specified by “filename” in dhcpd.conf. It uses pxelinux.cfg and boot.txt that we will shortly create to display a menu with options on which kernel to boot. We define two options. The default is ubuntu and it is loaded after 3 seconds. The other, admin, has proven helpful if you want to install new programs. Being the admin you don’t want to install them in a per User layer, but in the shared base. To go into admin mode, hit some key early at network boot, type “admin” and hit enter.mkdir -p /tftpboot/static/boot/pxelinux.cfg
emerge syslinux
cp /usr/share/syslinux/pxelinux.0 /tftpboot/static/boot
nano /tftpboot/static/boot/pxelinux.cfg/default
DISPLAY boot.txt
DEFAULT ubuntu
LABEL ubuntu
kernel /vmlinuz
append initrd=inittrd.img rw root=/dev/nfs ip=dhcp
LABEL admin
append initrd=initrd.img rw root=/dev/nfs nfsroot=10.11.2.2:/tftpboot/static/root ip=dhcp –
PROMPT 1
TIMEOUT 3
nano /tftpboot/static/boot/boot.txt
- Boot Menu -
=============
ubuntu
admin
#boot admin for refsys administration
rm /etc/udev/rules.d/70-persistent-net.rules
exit;reboot (now boot from harddisk)
INSTALL CLIENT SYSTEM AND CREATE NETBOOT KERNEL AND RAMDISK
Now you have to fetch a working ubuntu installation and stuff it into your shared nfs folder. There are several possibilities to get this done. I do it, by installing a normal ubuntu into a virtual machine and then use “tar” to create an archive, containing the whole filesystem, copy it to the server and unpack. The command is: $tar -cpP –absolute-names -f stage-ubuntu.tar /copy stage-ubuntu.tar to server
on the server in the nfsroot: #tar -xpvf stage-ubuntu.tar
You could also try and run debootstrap and chroot. Configure your system as you wish. Install any packages. Do some customization. Whatever pleases you. When you’re done, we have to create a netboot ramdisk. Ubuntu comes with a nice tool to help us, create it. The ubuntu kernel and the created ramdisk will then be stored on the netboot server. We also rearrange the filesystem on the server a bit, since initially we created an extra partition for /home of the guest os. We configure /etc/fstab and our network interfaces $cp /boot/vmlinuz-`uname -r` /root/vmlinuz
nano /etc/initramfs-tools/initramfs-conf
modules=netboot
boot=nfs
$mkinitramfs -o /root/initrd.img
remember to set initramfs-conf back to
modules=most
boot=local
when finished run the tar command as explained above: #tar -cpP –absolute-names -f stage-ubuntu.tar /
copy your stage to your nfsroot /tftpboot/static/root and unpack, using:
$tar -xpvf stage-ubuntu.tar
$mv /tftpboot/static/root/home/* /tftpboot/static/home
$mv /tftpboot/static/root/root/* /tftpboot/static/boot
cp /etc/resolv.conf /tftpboot/static/root/etc
nano /tftpboot/static/root/etc/fstab
/dev/nfs / nfs rsize=8192,wsize=8192,noatime,async 0 0
192.168.5.1:/tftpboot/static/home/ /home nfs rsize=8192,wsize=8192,noatime,async 0 0
none /proc proc nodev,noexec,nosuid 0 0
none /tmp tmpfs defaults 0 0
nano /tftpboot/static/root/etc/network/interfaces
auto lo
iface lo inet loopback
#auto eth0
iface eth0 inet manual #this is important, otherwise system wont boot
SETUP FOLDERS AND MOUNTS AND EXPORTS ON THE SERVER
I provide a little script here, that you can use as an idea of how to setup your per-client root paths. We create a directory tree for every client machine that contains folders “root” and “tmpfs”, then call aufs to:set /tftpboot/static/root as the read-only layer (we discussed this)
set /tftpboot/dynamic/<ip>/tmpfs as the write layer
and show it on /tftpboot/dynamic/<ip>/root
We then declare /tftpboot/dynamic/<ip>/root as an nfs export
—–snip——
#!/bin/bash
#get param
IP=$1
#create dirs
mkdir -p /tftpboot/dynamic/10.11.4.$IP/root
mkdir -p /tftpboot/dynamic/10.11.4.$IP/tmpfs
#aufs mount -t aufs -o br=/tftpboot/dynamic/10.11.4.$IP/tmpfs=rw:/tftpboot/static/root=ro none /tftpboot
/dynamic/10.11.4.$IP/root
echo “/tftpboot/dynamic/10.11.4.$IP/root 10.11.0.0/16(rw,async,fsid=$IP,no_subtree_check,no_root_squash,no_all_squash,no_acl)” >> /etc/exports
exportfs -r
——snap—–
That done, we delete /tftpboot/static/root/etc/udev/rules.d/70-persistent-net.rules
$sudo rm /tftpboot/static/root/etc/udev/rules.d/70-persistent-net.rules
That’s because this file contains a static binding of a network adapter that can be problematic, given the fact that many clients with different network adapters will boot this system. Restart the services and it’s done.
$/etc/init.d/nfs restart
$/etc/init.d/in.tftpd restart
RUN A NETBOOT CLIENT AND ENJOY
Remember, for administration, you probably want to make use of the admin mode we named earlier.Author: Fabian Schütz
Debian from scratch on lvm2 and software raid
Debian does not use any custom “boot flags”, as Gentoo does, where you specify “dolvm, domdadm” as kernel parameters in grub configuration, but offers a tool to create a ramdisk, suited for the job. $update-initramfs can be called via command line, but first, some settings need to be made. update-initramfs will read /etc/mdadm/mdadm.conf to retrieve configuration so the raid arrays can be assembled. Basically you want to have something like this
———snip———-
ARRAY /dev/md0 metadata=1.0 devices=/dev/sda1,/dev/sdb1
ARRAY /dev/md1 metadata=1.1 devices=/dev/sda2,/dev/sdb2
———snap———-
in your mdadm.conf.
Further more, update-initramfs will look into /etc/fstab and /boot/grub/menu.lst to gather information about the root device/partition. I fiddled a bit here, but in the end, it seemed, that devices, whose paths contain “mapper” are indentified as logical volumes, thus enabling lvm on boot. I tried
——————–
menu.lst
kernel /vmlinuz-xxx root=/dev/system/root ro quiet
——————–
fstab
/dev/system/root / ext3 defaults 0 1
——————–
first, but that didn’t work. So i put it this way
——————–
menu.lst
kernel /vmlinuz-xxx root=/dev/mapper/system-root ro quiet
——————–
fstab
/dev/mapper/system-root / ext3 defaults 0 1
——————–
and my system would boot. With “system” being my volume group, you basically need a path of this scheme: /dev/mapper/[volume-group]-[volume] in both /etc/fstab and /boot/grub/menu.lst.
That done, run either $update-initramfs -u if you want to update an existing ramfs or create a new one, using $update-initramfs -c -k . The version-label can is only a name and can entirely be made up.
Author: Fabian Schütz
Wednesday, October 6, 2010
New version of HTF: now backwards-compatible with HUnit
I’ve just uploaded version 0.5.0.0 of the Haskell Test Framework (HTF) to hackage. The new version allows for backwards-compatibility with the HUnit library. So, for example, say you have the following existing HUnit test:
test_fac =
do assertEqual "fac 0" 1 (fac 0)
assertEqual "fac 3" 6 (fac 3)
To let the HTF collect your unit tests automatically, you just need to add the following line at the top of your source file:
{-# OPTIONS_GHC -F -pgmF htfpp -optF --hunit #-}
The pragma above specifies that the source file should be run through HTF’s preprocessor htfpp in HUnit-backwards-compatibility mode. The preprocessor attaches precise location information to all assertions and collects all unit tests and all QuickCheck properties in a fresh variable called allHTFTests.
If you start with your unit tests from scratch, you should leave out the -optF --hunit flag because it releaves you from providing location information such as "fac 0" and "fac 1" for your testcases by hand. The pragma should then look as follows:
{-# OPTIONS_GHC -F -pgmF htfpp #-}
See the HTF tutorial for more information.
Thanks to Magnus Therning, who convinced me to add the HUnit-backwards-compatibility layer to the HTF.
Author: Stefan Wehr
Tuesday, September 28, 2010
Speeding up your cabal builds - Part II
Last time, I blogged about how linking your binaries against an internal library might speed up your cabal builds. This time, I show how you can avoid building certain binaries at all.
In our company, we work at a rather large Haskell project. The cabal file specifies more then ten binaries, so it takes rather long to build all of them. But often, you only need one or two of these binaries, so building them all is a waste of time.
Unfortunately, cabal does not allow you to build only a subset of your binaries. One workaround is the set the buildable flag in your .cabal file to false for the binaries you don’t want to build. However, this approach is rather unflexible because you need to edit the .cabal file and do a cabal configure after every change.
The solution I present in this article allows you to specify the binaries to build as arguments to the cabal build command. For example, if you want to build only binary B, you invoke cabal as cabal build B and cabal only builds binary B.
To get this working, all you need to do is writing a custom Setup.hs file:
import Data.List
import System.Exit
import Control.Exception
import Distribution.Simple
import Distribution.Simple.Setup
import Distribution.PackageDescription hiding (Flag)
import Distribution.PackageDescription.Parse
import Distribution.Verbosity (normal)
_CABAL_FILE_ = "DociGateway.cabal"
-- enable only certain binaries (specified on the commandline)
myPreBuildHook :: Args -> BuildFlags -> IO HookedBuildInfo
myPreBuildHook [] flags = return emptyHookedBuildInfo
myPreBuildHook args flags =
do let verbosity = case buildVerbosity flags of
Flag v -> v
NoFlag -> normal
descr <- readPackageDescription verbosity _CABAL_FILE_
let execs = map fst (condExecutables descr)
unbuildableExecs = execs \\ args
mapM_ (checkExistingExec execs) args
putStrLn ("Building only " ++ intercalate ", " args)
return (Nothing, map (\e -> (e, unbuildable)) unbuildableExecs)
where
unbuildable = emptyBuildInfo { buildable = False }
checkExistingExec all x =
if not (x `elem` all)
then do putStrLn ("Unknown executable: " ++ x)
throw (ExitFailure 1)
else return ()
main = defaultMainWithHooks $ simpleUserHooks { preBuild = myPreBuildHook }
That’s all! Don’t forget the set the Build-Type in your .cabal file to Custom. I’ve tested this approach with cabal-install version 0.8.2, using version 1.8.6 of the Cabal library.
Happy hacking and have fun!
Author: Stefan Wehr
Thursday, August 26, 2010
Speeding up your cabal builds
Every waited too long for your cabal builds to finish? If that’s because you have multiple executable sections in your .cabal file, then there might be a solution.
By default, cabal rebuilds all relevant object files for each executable in separation. In other words, object files are not shared between executables. So if you have n executables and m source files, then cabal needs n * m compilation steps plus n link steps to rebuild the executables, no matter whethe any source file contributes to multiple executables.
Starting with cabal 1.8, there is a better solution, provided your executables have some source files in common. In this case, you might build a library from these common source files and then link the executables against the library. In the example above, if all n executables use the same set of m source files, then you end up with m compilation steps plus n + 1 link steps. Sounds good, doesn’t it?!
Here is a simple .cabal file that demonstrates how linking against an internal library works:
Name: test Version: 0.1 Synopsis: test package for linking against internal libraries Author: Stefan Wehr Build-type: Simple Cabal-version: >=1.8 -- IMPORTANT Library Hs-source-dirs: lib -- IMPORTANT Exposed-modules: A Build-Depends: base >= 4 Executable test-exe Build-depends: base >= 4, test, -- link against the internal library Main-is: Main.hs -- imports A Hs-source-dirs: prog -- IMPORTANT
There are some things to consider:
- The Cabal-Version must be greater or equal 1.8.
- The library and the executable must not use common source directories, otherwise the compiler does not pick the library but recompiles the source files.
- The library must be mentioned in the Build-depends of the executable
Running cabal build now gives the following output:
Building test-0.1... [1 of 1] Compiling A ( lib/A.hs, dist/build/A.o ) Registering test-0.1... [1 of 1] Compiling Main ( prog/Main.hs, dist/build/test-exe/test-exe-tmp/Main.o ) Linking dist/build/test-exe/test-exe ...
No rebuilding of A when compiling Main!!!
This feature of cabal isn’t mentioned in the manual, at least I didn’t find it. Further, there seems to be no changelog for cabal. I found out about this feature by browsing the bug tracker for cabal. Is there a better way to get informed of new features of cabal?
Note: I successfully tested this with cabal-install version 0.8.2 (cabal library 1.8.0.4). I couldn’t get it to work with cabal-install version 0.8.0.
Author: Stefan Wehr
