Python - 常用Packages

1. PyInstaller - 打包Python程序

http://www.pyinstaller.org/

$ pip install pyinstaller
$ pyinstaller yourprogram.py

image.png

2. Locust - 测试工具，可完成压力测试、功能测试等

https://docs.locust.io/en/stable/quickstart.html

• test script

from locust import HttpLocust, TaskSet, task

class UserBehavior(TaskSet):
    def on_start(self):
        """ on_start is called when a Locust start before any task is scheduled """
        self.login()

    def on_stop(self):
        """ on_stop is called when the TaskSet is stopping """
        self.logout()

    def login(self):
        self.client.post("/login", {"username":"ellen_key", "password":"education"})

    def logout(self):
        self.client.post("/logout", {"username":"ellen_key", "password":"education"})

    @task(2)
    def index(self):
        self.client.get("/")

    @task(1)
    def profile(self):
        self.client.get("/profile")

class WebsiteUser(HttpLocust):
    task_set = UserBehavior
    min_wait = 5000
    max_wait = 9000

• run for test

$ locust -f locust_files/my_locust_file.py --host=http://example.com

• open http://127.0.0.1:8089 to start and analyse test results

image.png

`

3. luigi - 任务调度系统

3.1. 概述

Luigi is a Python (2.7, 3.6, 3.7 tested) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

image.png

image.png

3.2. demo

3.2.1. hello world

• hello_world.py

import luigi


class HelloWorldTask(luigi.Task):
    task_namespace = 'examples'

    def run(self):
        print("{task} says: Hello world!".format(task=self.__class__.__name__))


if __name__ == "__main__":
    luigi.run(['examples.HelloWorldTask', '--workers', '1', '--local-scheduler'])

• run

$ python hello_world.py
...
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 ran successfully:
    - 1 examples.HelloWorldTask()

This progress looks :) because there were no failed tasks or missing dependencies

===== Luigi Execution Summary =====

3.2.2. foo-bar

• foo.py

from __future__ import print_function
import time
import luigi


class Foo(luigi.WrapperTask):
    task_namespace = 'examples'

    def run(self):
        print("Running Foo")

    def requires(self):
        for i in range(10):
            yield Bar(i)


class Bar(luigi.Task):
    task_namespace = 'examples'
    num = luigi.IntParameter()

    def run(self):
        time.sleep(1)
        self.output().open('w').close()

    def output(self):
        """
        Returns the target output for this task.
        :return: the target output for this task.
        :rtype: object (:py:class:`~luigi.target.Target`)
        """
        time.sleep(1)
        return luigi.LocalTarget('/tmp/bar/%d' % self.num)


if __name__ == '__main__':
    luigi.run()

• run

$ python foo.py examples.Foo
...
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 complete ones were encountered:
    - 1 examples.Foo()

Did not run any tasks
This progress looks :) because there were no failed tasks or missing dependencies

===== Luigi Execution Summary =====

3.3. 参考

• luigi examples
  https://github.com/spotify/luigi/tree/master/examples

• luigi documents
  https://luigi.readthedocs.io/en/stable/tasks.html

• 浅谈工作流调度系统
  https://blog.csdn.net/hsg77/article/details/53409076