HPR2948: Testing with Haskell

Intro

I have liked writing automated tests for a long time, so it’s not a surprise that I end up writing them in Haskell too. This is very broad topic, so this episode only scratches the surface.

HSpec

HSpec is testing framework that automatically detects tests, like most of the modern systems. It supports hierarchies, so one can organize tests by feature for example.

spec :: Spec
spec = do
    describe "Very important feature" $ do
        it "Execution should be error free" $ do
            ...

        it "Flux capacitors can be charged" $ do
            ...

    describe "Somewhat less important feature" $ do
        ...

Unit test

Unit test tests a single case with fixed set of inputs. With pure functions these are a pleasure to write as they’re really just data in, data out, verify results. Below is two examples:

spec :: Spec
spec = do
    describe "Markov chain configuration" $ do
        it "Adding new starting element to empty configuration creates item with frequency of 1" $ do
            let config = addStart ("AA" :: DT.Text) emptyConfig
            config ^? (configStartsL . _head . itemFreqL) `shouldBe` Just 1
            config ^? (configStartsL . _head . itemItemL . _Just) `shouldBe` Just "AA"

        it "Adding same element twice to empty configuration creates item with frequency of 2" $ do
            let config = addStart "AA" $
                         addStart ("AA" :: DT.Text) emptyConfig
            config ^? (configStartsL . _head . itemFreqL) `shouldBe` Just 2
            config ^? (configStartsL . _head . itemItemL . _Just) `shouldBe` Just "AA"

Both are for testing configuring markov chains. First one checks that adding a starting element in empty configuration results correct item with correct weight being added. Second checks that adding same starting element twice results weight of 2.

Both tests use lenses for reading nested data structure. Episode doesn’t cover them much at all, as it’s enough to know that (configStartsL . _head . itemFreqL) focuses on starting elements of configuration, selects first item of the list and then selects frequency of that item. Lenses can also be used for modifying data and they don’t have to focus on only one element.

Unit tests are easy enough to write, they verify single thing about the unit being tested and are usually super fast to run and not error prone.

Property based test

Property based tests are used to check that a certain property holds with randomly generated input parameters. I’m using HSpec as testing framework and QuickCheck as tool for generating test data:

spec :: Spec
spec = do
    describe "planets" $ do
        describe "food" $ do
            it "food requirement for positive amount of population is more than zero" $ do
                forAll positivePopulation $ \x -> foodRequirement x  > RawResource 0

            it "food base production for farms is equal or greater than their amount" $ do
                forAll someFarms $ \x -> (sum (fmap foodBaseProduction x)) > (RawResource $ length x)

Above we have two tests. First one checks that with any non-zero population, foodRequirement is greater than 0. Second one check that with any positive amount of farm, foodBaseProduction is greater than amount of the farms.

positivePopulation is Generator, that is used by QuickCheck to generate random data for testing. Its definition is shown below:

singlePopulation :: Gen PlanetPopulation
singlePopulation = do
    let aPlanetId = toSqlKey 0
    let aRaceId = toSqlKey 0
    aPopulation <- arbitrary `suchThat` \x -> x > 0
    return $ PlanetPopulation aPlanetId aRaceId aPopulation

positivePopulation :: Gen [PlanetPopulation]
positivePopulation = do
    k <- arbitrary `suchThat` \x -> x > 0
    vectorOf k singlePopulation

Generated data can be really simple or very complex. Generating complex data is often convenient to break into smaller steps and write generators for them.

Property based tests are somewhat harder to write than unit tests, but they can potentially cover edge cases that might otherwise not been discovered.

Working with database

All tests shown so far have been testing pure code, that is, code that is data in, data out. When database is introduced, things get more complicated. Suddenly there’s much more possibilities for errors. Below is an example of such a test:

spec :: Spec
spec = withApp $ do

    describe "Status handling"  $ do
        describe "Planet statuses"  $ do
            it "Expired planet statuses are removed and news created" $ do

                sId <- runDB $ insert $ StarSystem
                        { starSystemName = "Aldebaraan"
                        , starSystemCoordX = 10
                        , starSystemCoordY = 20
                        , starSystemRulerId = Nothing
                        }

                fId <- runDB $ insert $ Faction
                        { factionName = "Star lords"
                        , factionHomeSystem = sId
                        , factionBiologicals = 10
                        , factionMechanicals = 10
                        , factionChemicals = 10
                        }

                pId1 <- runDB $ insert $ Planet
                        { planetName = "New Earth"
                        , planetPosition = 3
                        , planetStarSystemId = sId
                        , planetOwnerId = Just fId
                        , planetGravity = 1.0
                        , planetRulerId = Nothing
                        }

                _ <- runDB $ insert $ PlanetStatus
                        { planetStatusPlanetId = pId1
                        , planetStatusStatus = GoodHarvest
                        , planetStatusExpiration = Just 20201
                        }

                let status = Simulation 20201
                _ <- runDB $ insert status

                news <- runDB $ removeExpiredStatuses (simulationCurrentTime status)

                statuses <- runDB $ selectList [ PlanetStatusPlanetId ==. pId1 ] []
                loadedNews <- runDB $ selectList [] [ Asc NewsDate ]

                liftIO $ statuses `shouldSatisfy` (\x -> length x == 0)
                liftIO $ news `shouldSatisfy` (\x -> length x == 1)
                liftIO $ loadedNews `shouldSatisfy` (\x -> length x == 1)

There’s a lot more code that had to be written for this test and majority of it is for setting up database state. The test if for ensuring that when good harvest boost expires, it is removed from database and respective news article is created.

These kinds of tests have a lot more code and are much more slower to run because of the communication with a database. There’s also more cases where something can go wrong. But in the end, these kinds of tests are needed if one wants to verify that interaction with database is working as planned.

Testing API

Last example is about testing REST API. There are two tests, where the first one is checking that proper access control is in place and second one checks that pending messages are correctly retrieved.

spec :: Spec
spec = withApp $ do
    describe "Message handling" $ do
        it "unauthenticated user can't access messages" $ do
            _ <- get ApiMessageR
            statusIs 401

        it "pending messages are loaded" $ do
            (pId, fId) <- setupPerson
            _ <- runDB $ insert $ researchCompleted 25250 fId HighSensitivitySensors
            user <- createUser "Pete" (Just pId)
            authenticateAs user
            _ <- get ApiMessageR
            resp <- getResponse
            let jsonM = join (decode <$> simpleBody <$> resp) :: Maybe Value

            assertEq "message tag"
                     (jsonM ^? (_Just . _Array . _head . key "tag" . _String))
                     (Just "ResearchCompleted")
            assertEq "star date"
                     (jsonM ^? (_Just . _Array . _head . key "starDate" . _Integer))
                     (Just 25250)
            assertEq "technology"
                     (jsonM ^? (_Just . _Array . _head . key "contents" . key "Technology" . _String))
                     (Just "HighSensitivitySensors")

            statusIs 200

Here extra complication is created by the fact that many features of the system are behind authentication and authorization. Luckily Yesod comes with helper function authenticateAs, that allows code to authenticate when system is running in development mode.

These test are even slower than any of the previous ones, but on the other hand, they test whole chain from user interaction to database and back.

In closing

There’s lots of things that I couldn’t cover in such a short time, like various types of tests: UI testing, performance testing, security testing, long running testing…, the list goes on and on. But hopefully this episode gave you ideas what kinds of tests one can write and how to get started doing so using Haskell.

Best way to reach me is email or at fediverse, where I’m Tuula@mastodon.social.

Comments

More Information...