Gatling Version 2.1.7

Gatling Version 2.1.7

Gatling only have JSON, CSV, JDBC and some other support. So I wrote the XML support myself.

1 Setup the Gatling Project
plugins.sbt will enable all the plugins
addSbtPlugin("io.gatling" % "gatling-sbt" % "2.1.7")
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")

build.properties SBT version is update to date
sbt.version=0.13.8

.sbtopts to make the Memory Ok for Large Testing Data File
-J-Xmx1G
-J-XX:+UseConcMarkSweepGC
-J-XX:+CMSClassUnloadingEnabled
-J-Xss2M -Duser.timezone=GMT

Copy and use the default application.con, gatling.conf, logback.xml.

My base class to Prepare the Perf Env in PerformanceTesterEnvironment.scala
package com.sillycat.performance.base
import com.typesafe.config.ConfigFactory
trait PerformanceTesterEnvironment {

  val config = ConfigFactory.load

  val env = "env." + config.getString("build.env")

  private def path(s: String*) = s.mkString(".")

  def envStr(s: String): String = {
    path(env, s)
  }

  val rampSeconds = 1

  val rampNumOfUsers = config.getInt(envStr("ramp.user.num"))

  val requestsPerUser = config.getInt(envStr("requests.per.user"))

  val classifierHost = config.getString(envStr("classifier.host"))

}

A typical GET JSON API Simulation, ClassifierGetIndustrySimulation.scala
package com.sillycat.performance.classifier

import com.sillycat.performance.base.PerformanceTesterEnvironment
import com.sillycat.performance.feeder.CustomerFeederSupport

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class ClassifierGetIndustrySimulation extends Simulation with PerformanceTesterEnvironment with CustomerFeederSupport{

  val httpConf = http
    .baseURL(classifierHost)
    .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") // Here are the common headers
    .doNotTrackHeader("1")
    .acceptLanguageHeader("en-US,en;q=0.5")
    .acceptEncodingHeader("gzip, deflate")
    .userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0")

  val headers =
    Map("Accept" -> "application/json, text/javascript, */*; q=0.01",
      "Keep-Alive" -> "180",
      "X-Requested-With" -> "XMLHttpRequest",
      "Content-type" -> "application/json")

  //val jobTitleDescJSONFeed = jsonFile("job.title.desc.json").random
  val jobTitleDescXMLFeed = xmlFile("job.title.desc.xml.small").random

  val classifyJob2Industry = exec(http("classifyJob2Industry")
    .get("/industry")
    .headers(headers)
    .queryParamMap(Map("title"->"${title}","description" -> "${description}", "format" -> "json"))
    .check(status.is(200))
    .check(bodyString.exists))

  val multiScn = scenario("Classify Industry")
    .repeat(requestsPerUser){ feed(jobTitleDescXMLFeed).exec(classifyJob2Industry) }

  setUp(
    multiScn.inject(rampUsers(rampNumOfUsers) over (1 seconds))
  ).protocols(httpConf)

}

2 Add Support to XML for Gatling
package com.sillycat.performance.feeder

import io.gatling.core.config.Resource
import io.gatling.core.feeder._
import io.gatling.core.validation.{Failure, Success, Validation}

trait CustomerFeederSupport {

  def xmlFile(fileName: String): RecordSeqFeederBuilder[Any] = xmlFile(Resource.feeder(fileName))

  def xmlFile(resource: Validation[Resource]): RecordSeqFeederBuilder[Any] =
    feederBuilder(resource)(XMLFeederFileParser.parse)

  def feederBuilder[T](resource: Validation[Resource])(recordParser: Resource => IndexedSeq[Record[T]]): RecordSeqFeederBuilder[T] =
    resource match {
      case Success(res)     => RecordSeqFeederBuilder(recordParser(res))
      case Failure(message) => throw new IllegalArgumentException(s"Could not locate feeder file; $message")
    }

}

package com.sillycat.performance.feeder

import java.io.InputStream
import java.net.URL

import io.gatling.core.feeder.Record

import scala.collection._
import io.gatling.core.config.Resource
import io.gatling.core.util.IO._

import scala.xml.{NodeSeq, XML}

object XMLFeederFileParser {

  def parse(resource: Resource): IndexedSeq[Record[Any]] =
    withCloseable(resource.inputStream) { is =>
      stream(is).toVector
    }

  def url(url: String): IndexedSeq[Record[Any]] =
    withCloseable(new URL(url).openStream) { is =>
      stream(is).toVector
    }

  def stream(is: InputStream): Iterator[Record[Any]] = {

    val xml = scala.xml.XML.load(is)

    (xml \ "JOB").iterator.collect {
      case job:NodeSeq => {
        Map("title" -> (job \\ "TITLE").text, "description" -> (job \\ "DESC").text).toMap
      }
      case _ => {
        throw new IllegalArgumentException("Root element of XML feeder file isn't an array")
      }
    }

  }

}

Tips
1 Codes to find the mismatched Tag in XML
package com.sillycat.performance

import scala.io.Source

/**
  * Created by carl on 12/2/15.
  */
object MainTest extends App{

  println("start calculating...")

  var line_num = 0
  var flag = 0
  for(line <- Source.fromFile("/Users/carl/company/code/api-performance/user-files/data/classify.industry.xml.large").getLines()){
    line_num = line_num + 1
    if(line.contains("<JOB ")){
      flag = flag + 1
    }
    if(line.contains("</JOB>")){
      //println(line)
      flag = flag - 1
    }
    if(flag > 1){
      println(line_num + " " + flag)
      flag = 0
    }
  }
}

2 XML Validation Error
Exception:
org.xml.sax.SAXParseException; lineNumber: 136589; columnNumber: 10; The content of elements must consist of well-formed character data or markup.
Exception
org.xml.sax.SAXParseException; lineNumber: 263; columnNumber: 49; The entity name must immediately follow the '&' in the entity reference.

Solution:
sublime - [Goto] - [Goto Line]
https://support.microsoft.com/en-us/kb/308060

Find the line number, lookup the special characters mapping, change < to &lt; or > to &gt;

3 Unicode Validation Error
Exception
org.xml.sax.SAXParseException; lineNumber: 1051001; columnNumber: 532; An invalid XML character (Unicode: 0x19) was found in the element content of the document.

Solution:
Right now, I just delete that.

4 HTTP Length Issue
Exception
org.jboss.netty.handler.codec.frame.TooLongFrameException: An HTTP line is larger than 4096 bytes.
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.readLine(HttpMessageDecoder.java:670) ~[io.netty.netty-3.10.4.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:184) ~[io.netty.netty-3.10.4.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102) ~[io.netty.netty-3.10.4.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[io.netty.netty-3.10.4.Final.jar:na]

Solution:
We should use POST via form data instead for large length of params.

References:
http://gatling.io/docs/2.1.6/extensions/sbt_plugin.html
https://github.com/gatling/gatling-sbt-plugin-demo